If 'ConnectionLog' and/or 'SessionLog' are set to verbose or higher, I expect the following error shown in the maillog, if assp is unable to close a tcp-socket to 'smtpDestination'.
error: unable to close Socket SOCKET - PERLERROR - SYSTEMERROR Notice: - the assp development version you use is 18 builds behind the current build - ASSP 2.8.2(26253) - Perl 5.26.1 status: Release Date: September 2017 Active Support Ended: May 2019 Security Support Ended: May 2020 Current Status: Unsupported and obsolete - ASSP 2.8.2(24261) is used by only 3 instances (including yours) worldwide - so I don't expect any feedback for this build 1. Is there a known issue with smtpDestination connection handling/closing in this version or fixed in a later release? please read the changelog.txt by your self! so I could skip the copy and paste. build 24261 2024-09-17 fixed: - related to https://assp.thockar.com/forum/viewtopic.php?t=3700 If enableINET6 was used, binding to a listener after an assp restart failed on some systems, because assp used the deprecated IO::Socket::INET option 'Reuse' also in calls to IO::Socket::IP, where this option is not supported. Now assp uses the 'ReuseAddr' option instead, which is supported in both modules. added: - related to https://assp.thockar.com/forum/viewtopic.php?t=3700 If $disable_SO_REUSEPORT is set to 0 or 2 assp tries to use the SO_REUSEPORT socket option our $disable_SO_REUSEPORT = 1; # (0/1/2) disable the SO_REUSEPORT socket option - there is no assp version which ever used this option # notice: windows never has this socket option - so leave this value at 1 # 0 - do not disable - try it, but if not supported by the OS it is not used and a load warning is produced for the module 'Socket' # 1 - disable this socket option and do not try to use it # 2 - do not disable - try it, but if not supported by the OS it is not used and silently ignored You may try to play around with this value, if you get unexpected errors for assp listeners while bind or reuse. !!! the related code is currently untouched since build 24261 2. Is there a config option to disable backend connection reuse/pooling (if that's what's happening here), or to force ASSP to close backend connections immediately after each transaction rather than potentially holding them open for reuse? see above ASSP needs and tries to close every connection. REUSE.... options are used by assp at creation time of a socket listener, and should be handled by the underlying system components (kernel, libc ...) ASSP uses the socket it gets from the system at socket creation time. It uses REUSEADDR for every (but only) listener - usage of REUSEPORT depends on config ($disable_SO_REUSEPORT) and system support. Connections to the 'smtpDestination' backend are tcp client connections, and should be not affected by any of these parameters. I just had a look in to my "small" knowledgebase and found a nice explanation of a very special tcp CLOSE_WAIT case: "The connection might also go to CLOSE_WAIT even if the server didn't yet invoke accept() on the socket and the client already close()" So - if your backenserver gets the connection, but does'nt do a sock->accept() for any reason, it may happen that the connection stays in CLOSE_WAIT at the assp system - even assp closed the socket clearly! It is currently unclear to me what's going on in this case (e.g. the assp system sends a RST instead of a FIN ?? - the FIN to the backend is the missing part here) If this (or anything similar) happens, you'll not see the assp error message in the log "error: unable to close Socket SOCKET - PERLERROR - SYSTEMERROR" because I expect the system returns success on the assp close() call. It makes sense to me, that SSL connections are not affected. SSL connections are mostly completely and hard closed at both sites - for security reasons. btw: did you change/update/install anything on any of the involved systems ? This is just a shot into the dark. Checking both, the backend log and assp log (time based) using some increased log levels may lead into some more brightness. I'm sure, I can't help you any further. Possibly someone else is able to bring some light into this. 3. Has anyone else seen CLOSE_WAIT accumulation specifically on the smtpDestination leg rather than the client-facing leg? there is no related issue reported for build 24261 and all higher builds Thomas Von: "Robert Ellsworth" <[email protected]> An: "For Users of ASSP" <[email protected]> Datum: 15.09.2026 16:00 Betreff: [Assp-user] [Bug Report] Persistent CLOSE_WAIT socket leak on smtpDestination (backend) connections - ASSP 2.8.2(24261) Subject: [Bug Report] Persistent CLOSE_WAIT socket leak on smtpDestination (backend) connections - ASSP 2.8.2(24261) Hi all, I'm seeing a reproducible file descriptor leak on our ASSP installation that eventually leads to "Too many open files" errors and worker crashes if left unaddressed. I've isolated it fairly precisely and wanted to share the evidence in case it's a known issue or others are hitting it silently. ## Environment - ASSP version: 2.8.2(24261) - OS: Debian/Ubuntu (Linux), Perl 5.26.1 - Run mode: systemd-managed daemon (Type=forking, PIDFile), ulimit raised to 65536 (soft and hard) to rule out an undersized limit as the cause - smtpDestination:=192.168.1.54:25|SSL:192.168.1.54:465 (backend is Zimbra, both plaintext :25 and SSL :465 destinations configured) ## Symptom Open file descriptor count on the assp.pl process climbs steadily over several hours, with no plateau, until it approaches the ulimit ceiling and worker threads start failing with: Worker_X accept to client failed IO::Socket::INET=GLOB(...) (timeout: 2 s) : Too many open files main exception: error: AsspSelfLoader is unable to load code from file .../sl-cache/main-resetFH.sl - Too many open files This has now happened on multiple separate occasions (Aug 28, Sep 1, Sep 4, and again this morning, Sep 15), each following the same growth pattern: flat baseline FD count (~40-50) for hours, then a sustained climb beginning at an arbitrary point in the day and continuing until intervention. ## Isolating the leak Using `lsof -p <pid>`, I confirmed the growth is entirely in socket file descriptors, not regular files, disk cache handles, or DNS-related opens (REG/CHR/DIR counts stay flat throughout): 12:13:47 86 REG 46 IPv4 7 CHR 5 sock 2 DIR 12:28:47 86 REG 115 IPv4 7 CHR 11 sock 2 DIR 12:43:48 87 REG 293 IPv4 7 CHR 50 sock 2 DIR 12:58:48 86 REG 361 IPv4 92 sock 7 CHR 2 DIR 13:13:49 88 REG 536 IPv4 126 sock 7 CHR 2 DIR 13:43:56 86 REG 668 IPv4 255 sock 7 CHR 2 DIR Breaking down socket state with `lsof -p <pid> -i` during an active leak event this morning: 335 CLOSE_WAIT 28 ESTABLISHED 12 LISTEN CLOSE_WAIT dominates - meaning the remote peer has already closed its end (sent FIN) but ASSP has not called close() on its side. Breaking the CLOSE_WAIT sockets down by remote host: 246 our.backend.mailserver <- our smtpDestination backend 72 mailsrv.graphic.com.gh 2 mail-qv2-f12.google.com 1 (various other single-connection external senders - normal background noise) The overwhelming majority (73%+) are connections to our own backend/ destination server (Zimbra), not inbound client connections. Inspecting the port on these: perl 7243 root 46u IPv4 ... TCP assp.eomega.org:37038->our.backend.mailserver:smtp (CLOSE_WAIT) perl 7243 root 51u IPv4 ... TCP assp.eomega.org:43076->our.backend.mailserver:smtp (CLOSE_WAIT) ... (continues, ~295 similar entries) Nearly all of these are the plaintext `smtp` (port 25) destination, not the SSL (:465) destination - so this does not appear to be an SSL-shutdown issue, but rather plain-TCP backend connections that are not being closed after the delivery/proxy transaction completes. ## What this looks like to me It appears that when ASSP proxies a message through to smtpDestination and the backend (Zimbra) closes its end of the connection after completing the SMTP transaction, ASSP does not follow up with its own close() on that socket - leaving it in CLOSE_WAIT indefinitely. Over hours, under normal mail volume, this accumulates into the thousands and exhausts the process's file descriptor limit. ## What I've ruled out - Undersized ulimit: raised to 65536, leak still occurs, just takes longer to become fatal - DNS resolver file-open failures: present in logs but unrelated; these are a side-effect of FD exhaustion (resolver can't open /etc/protocols), not a cause - SSL socket shutdown handling: leak is concentrated on the plaintext :25 backend connection, not :465 - Spam-flood/PenaltyBox tempfail handling: this was our working theory after an earlier incident (leak did correlate with a spam sender getting tempfailed repeatedly), but this most recent capture shows the leak is overwhelmingly backend-connection-related regardless of that theory - possibly a contributing trigger, but not the core mechanism - tcp_fin_timeout / kernel-level socket reclaim: not applicable, since CLOSE_WAIT is governed entirely by the application, not any kernel timeout ## Questions for the list 1. Is there a known issue with smtpDestination connection handling/closing in this version or fixed in a later release? 2. Is there a config option to disable backend connection reuse/pooling (if that's what's happening here), or to force ASSP to close backend connections immediately after each transaction rather than potentially holding them open for reuse? 3. Has anyone else seen CLOSE_WAIT accumulation specifically on the smtpDestination leg rather than the client-facing leg? Happy to provide additional lsof/tcpdump captures, debug logging, or test config changes if it helps narrow this down. Currently mitigating with a raised ulimit + systemd auto-restart to avoid crashes while we get to the bottom of the actual leak. Thanks, Rob_______________________________________________ Assp-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/assp-user
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Assp-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/assp-user
