Re: Too many references: cannot splice
Sebastian,
On October 13th I sent a new email with subject "Too many references:
cannot splice" with a reference to a lklm post.
You will find more info. and a patch there. I can confirm the patch is
working for us.
Regards,
Luis Ugalde
On Fri, Dec 2, 2016 at 5:23 PM, Sebastian Köhler <s...@tyrion.de> wrote:
> Hello,
>
> multiple times per day one of our Dovecot servers has the problem that it
> suddenly refuses to accept new connections and then floods the logs with
> these messages:
>
> Dec 2 12:43:06 alfa3201 dovecot: pop3-login: Error: fd_send(pop3, 18)
> failed: Too many references: cannot splice
> Dec 2 12:43:07 alfa3201 dovecot: pop3-login: Error: fd_send(pop3, 18)
> failed: Too many references: cannot splice
> Dec 2 12:43:07 alfa3201 dovecot: imap-login: Error: fd_send(imap, 16)
> failed: Too many references: cannot splice
> Dec 2 12:43:07 alfa3201 dovecot: imap-login: Error: fd_send(imap, 16)
> failed: Too many references: cannot splice
>
>
> We have to either do a restart of the server or a doveadm kick on all
> users to make it reachable again. On my search for a solution I found a
> similar report from March[1] however without a fix.
>
>
> [1] http://dovecot.org/pipermail/dovecot/2016-March/103514.html
>
>
> dovecot -n
>
> # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.13 (7b14904)
> # OS: Linux 4.4.17-040417-generic x86_64 Debian 8.6
> auth_mechanisms = plain login
> debug_log_path = /var/log/dovecot-debug.log
> default_client_limit = 8003
> default_process_limit = 2000
> default_vsz_limit = 512 M
> disable_plaintext_auth = no
> listen = *
> login_greeting = [CENSORED] - ready.
> mail_location = maildir:~/Maildir:CONTROL=/opt/dovecot/control/%n-
> control:INDEX=/opt/dovecot/index/%n
> mail_plugins = " stats quota"
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
> }
> mailbox Spam {
> auto = subscribe
> special_use = \Junk
> }
> mailbox Sent {
> auto = subscribe
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> auto = subscribe
> special_use = \Trash
> }
> prefix = INBOX.
> separator = .
> }
> passdb {
> args = dovecot-%s
> driver = pam
> }
> plugin {
> quota = fs:%n(soft quota)
> sieve = ~/.dovecot.sieve
> sieve_dir = ~/sieve
> stats_command_min_time = 1 mins
> stats_domain_min_time = 12 hours
> stats_ip_min_time = 12 hours
> stats_memory_limit = 16 M
> stats_refresh = 30 secs
> stats_session_min_time = 15 mins
> stats_track_cmds = no
> stats_user_min_time = 24 hours
> }
> protocols = " imap pop3"
> service auth {
> unix_listener /var/spool/postfix/private/auth {
> group = postfix
> mode = 0666
> user = postfix
> }
> }
> service imap-login {
> inet_listener imaps {
> ssl = yes
> }
> }
> service imap-postlogin {
> executable = script-login /etc/dovecot/post-login/expire.sh
> }
> service imap {
> executable = imap imap-postlogin
> }
> service pop3-login {
> inet_listener pop3 {
> port = 110
> }
> inet_listener pop3s {
> port = 995
> ssl = yes
> }
> }
> service pop3-postlogin {
> executable = script-login /etc/dovecot/post-login/expire.sh
> }
> service pop3 {
> executable = pop3 pop3-postlogin
> }
> service quota-status {
> executable = quota-status -p postfix
> unix_listener /var/spool/postfix/private/quota {
> group = postfix
> mode = 0660
> user = postfix
> }
> }
> service stats {
> fifo_listener stats-mail {
> mode = 0666
> user = dovecot
> }
> }
> ssl_ca = ssl_cert = ssl_key = ssl_protocols = !SSLv2 !SSLv3
> userdb {
> driver = passwd
> }
> protocol imap {
> imap_capability = +XLIST +NAMESPACE METADATA QUOTA
> imap_idle_notify_interval = 1 mins
> mail_max_userip_connections = 500
> mail_plugins = " stats quota imap_quota imap_stats"
> }
> protocol pop3 {
> mail_max_userip_connections = 500
> mail_plugins = " stats quota"
> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
> }
>
Re: Too many references: cannot splice
Hi, I've been running the patched version for the last few days, and I can confirm that I have not seen any errors. I still have to check it with 4.x kernels, but it's working as expected with the current Debian Stable one. Regards, Luis ugalde. On Wed, Oct 26, 2016 at 12:59 PM, Timo Sirainen <t...@iki.fi> wrote: > On 26 Oct 2016, at 11:14, Luis Ugalde <forondar...@gmail.com> wrote: > > > > Hi, > > > > Could you please have a look at https://lkml.org/lkml/2016/2/2/538 and > see > > if this makes any sense to you? I've been checking kernel changes > > between linux_3.16.7 and linux_3.16.36, and this has popped out. Could > this > > be the reason for the "too many references" errors? > > Does the attached patch help? > > > > > > > Regards, > > > > Luis Ugalde. > > > > On Thu, Oct 13, 2016 at 3:47 PM, Luis Ugalde <forondar...@gmail.com> > wrote: > > > >> Hi, > >> > >> > >> A while ago I sent an email regarding these "*ETOOMANYREFS* Too many > >> references: cannot splice." that we've seen since Debian updated the > Jessie > >> kernel to > >> > >> 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > >> > >> while older kernels, like 3.16.0-4-amd64 #1 SMP Debian > >> 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. > >> > >> I was wondering if no one else is getting these errors, or if you know > any > >> workarounds that might probe useful, apart from downgrading the kernel. > >> > >> > >> I would say that the infrastructure we're running is quite standard, > with > >> directors balancing users to NFS backed dovecot servers. > >> > >> > >> Best regards, > >> > >> Luis Ugalde. > >> > >> > >> > >> > > >
Re: Too many references: cannot splice
Hi, Could you please have a look at https://lkml.org/lkml/2016/2/2/538 and see if this makes any sense to you? I've been checking kernel changes between linux_3.16.7 and linux_3.16.36, and this has popped out. Could this be the reason for the "too many references" errors? Regards, Luis Ugalde. On Thu, Oct 13, 2016 at 3:47 PM, Luis Ugalde <forondar...@gmail.com> wrote: > Hi, > > > A while ago I sent an email regarding these "*ETOOMANYREFS* Too many > references: cannot splice." that we've seen since Debian updated the Jessie > kernel to > > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > > while older kernels, like 3.16.0-4-amd64 #1 SMP Debian > 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. > > I was wondering if no one else is getting these errors, or if you know any > workarounds that might probe useful, apart from downgrading the kernel. > > > I would say that the infrastructure we're running is quite standard, with > directors balancing users to NFS backed dovecot servers. > > > Best regards, > > Luis Ugalde. > > > >
Too many references: cannot splice
Hi, A while ago I sent an email regarding these "*ETOOMANYREFS* Too many references: cannot splice." that we've seen since Debian updated the Jessie kernel to 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 while older kernels, like 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. I was wondering if no one else is getting these errors, or if you know any workarounds that might probe useful, apart from downgrading the kernel. I would say that the infrastructure we're running is quite standard, with directors balancing users to NFS backed dovecot servers. Best regards, Luis Ugalde.
Re: ETOOMANYREFS related errors
Hi, I've been running a Debian-Jessie backported 4.4 kernel for the last few days, and I can confirm the same issue happening with this kernel as well. Regards, Luis El vie., 11 mar. 2016 a las 9:01, Luis Ugalde (<forondar...@gmail.com>) escribió: > El vie., 11 mar. 2016 a las 2:08, Timo Sirainen (<t...@iki.fi>) escribió: > >> On 11 Mar 2016, at 03:48, Luis Ugalde <forondar...@gmail.com> wrote: >> > >> > Hi, >> > >> > I'm starting to see, on a pretty standard Debian Jessie installation, >> some >> > error messages that are apparently related to the ETOOMANYREFS errno. >> > >> > Firstly, the mail log shows this: >> > dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many >> references: >> > cannot splice >> >> Apparently because Linux thinks the same fd has been passed around >> recursively too many times: >> http://lkml.iu.edu/hypermail/linux/kernel/1101.0/01917.html >> >> But Dovecot doesn't pass it recursively. It's only passed once from >> pop3-login to pop3 process. >> >> > Is this something that Dovecot should be able to handle, or is it >> strictly >> > Debian/libc/MySillyMistake related? >> > >> > #uname -a >> > Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 >> > (2016-01-17) x86_64 GNU/Linux >> >> I wonder if there's a new kernel change that started detecting the >> recursion wrong. >> >> > Yes, It's started to happen with the latest kernels from the Stable > branch. This is the latest I have installed: > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > > Older Debian kernels are not showing up anything. This one, for example: > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 > > > Regards. >
Re: ETOOMANYREFS related errors
El vie., 11 mar. 2016 a las 2:08, Timo Sirainen (<t...@iki.fi>) escribió: > On 11 Mar 2016, at 03:48, Luis Ugalde <forondar...@gmail.com> wrote: > > > > Hi, > > > > I'm starting to see, on a pretty standard Debian Jessie installation, > some > > error messages that are apparently related to the ETOOMANYREFS errno. > > > > Firstly, the mail log shows this: > > dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many > references: > > cannot splice > > Apparently because Linux thinks the same fd has been passed around > recursively too many times: > http://lkml.iu.edu/hypermail/linux/kernel/1101.0/01917.html > > But Dovecot doesn't pass it recursively. It's only passed once from > pop3-login to pop3 process. > > > Is this something that Dovecot should be able to handle, or is it > strictly > > Debian/libc/MySillyMistake related? > > > > #uname -a > > Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 > > (2016-01-17) x86_64 GNU/Linux > > I wonder if there's a new kernel change that started detecting the > recursion wrong. > > Yes, It's started to happen with the latest kernels from the Stable branch. This is the latest I have installed: 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 Older Debian kernels are not showing up anything. This one, for example: 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 Regards.
ETOOMANYREFS related errors
Hi, I'm starting to see, on a pretty standard Debian Jessie installation, some error messages that are apparently related to the ETOOMANYREFS errno. Firstly, the mail log shows this: dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many references: cannot splice And then the login process fails: dovecot: pop3-login: Internal login failure (pid=34388 id=1) (internal failure, 1 successful auths): user=, method=PLAIN, rip=rip, lip=lip, session= Is this something that Dovecot should be able to handle, or is it strictly Debian/libc/MySillyMistake related? #uname -a Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux #dovecot --version 2.2.13 #dpkg -l | grep -E 'dovecot-core|libc6|linux-image' ii dovecot-core 1:2.2.13-12~deb8u1amd64 ii libc6:amd642.19-18+deb8u3amd64 ii libc6-i386 2.19-18+deb8u3amd64 ii linux-image-3.16.0-4-amd64 3.16.7-ckt20-1+deb8u3 amd64 Best Regards, Luis
