Re: Too many references: cannot splice
Sebastian, On October 13th I sent a new email with subject "Too many references: cannot splice" with a reference to a lklm post. You will find more info. and a patch there. I can confirm the patch is working for us. Regards, Luis Ugalde On Fri, Dec 2, 2016 at 5:23 PM, Sebastian Köhler <s...@tyrion.de> wrote: > Hello, > > multiple times per day one of our Dovecot servers has the problem that it > suddenly refuses to accept new connections and then floods the logs with > these messages: > > Dec 2 12:43:06 alfa3201 dovecot: pop3-login: Error: fd_send(pop3, 18) > failed: Too many references: cannot splice > Dec 2 12:43:07 alfa3201 dovecot: pop3-login: Error: fd_send(pop3, 18) > failed: Too many references: cannot splice > Dec 2 12:43:07 alfa3201 dovecot: imap-login: Error: fd_send(imap, 16) > failed: Too many references: cannot splice > Dec 2 12:43:07 alfa3201 dovecot: imap-login: Error: fd_send(imap, 16) > failed: Too many references: cannot splice > > > We have to either do a restart of the server or a doveadm kick on all > users to make it reachable again. On my search for a solution I found a > similar report from March[1] however without a fix. > > > [1] http://dovecot.org/pipermail/dovecot/2016-March/103514.html > > > dovecot -n > > # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.4.13 (7b14904) > # OS: Linux 4.4.17-040417-generic x86_64 Debian 8.6 > auth_mechanisms = plain login > debug_log_path = /var/log/dovecot-debug.log > default_client_limit = 8003 > default_process_limit = 2000 > default_vsz_limit = 512 M > disable_plaintext_auth = no > listen = * > login_greeting = [CENSORED] - ready. > mail_location = maildir:~/Maildir:CONTROL=/opt/dovecot/control/%n- > control:INDEX=/opt/dovecot/index/%n > mail_plugins = " stats quota" > namespace inbox { > inbox = yes > location = > mailbox Drafts { > auto = subscribe > special_use = \Drafts > } > mailbox Spam { > auto = subscribe > special_use = \Junk > } > mailbox Sent { > auto = subscribe > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > auto = subscribe > special_use = \Trash > } > prefix = INBOX. > separator = . > } > passdb { > args = dovecot-%s > driver = pam > } > plugin { > quota = fs:%n(soft quota) > sieve = ~/.dovecot.sieve > sieve_dir = ~/sieve > stats_command_min_time = 1 mins > stats_domain_min_time = 12 hours > stats_ip_min_time = 12 hours > stats_memory_limit = 16 M > stats_refresh = 30 secs > stats_session_min_time = 15 mins > stats_track_cmds = no > stats_user_min_time = 24 hours > } > protocols = " imap pop3" > service auth { > unix_listener /var/spool/postfix/private/auth { > group = postfix > mode = 0666 > user = postfix > } > } > service imap-login { > inet_listener imaps { > ssl = yes > } > } > service imap-postlogin { > executable = script-login /etc/dovecot/post-login/expire.sh > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3 { > port = 110 > } > inet_listener pop3s { > port = 995 > ssl = yes > } > } > service pop3-postlogin { > executable = script-login /etc/dovecot/post-login/expire.sh > } > service pop3 { > executable = pop3 pop3-postlogin > } > service quota-status { > executable = quota-status -p postfix > unix_listener /var/spool/postfix/private/quota { > group = postfix > mode = 0660 > user = postfix > } > } > service stats { > fifo_listener stats-mail { > mode = 0666 > user = dovecot > } > } > ssl_ca = ssl_cert = ssl_key = ssl_protocols = !SSLv2 !SSLv3 > userdb { > driver = passwd > } > protocol imap { > imap_capability = +XLIST +NAMESPACE METADATA QUOTA > imap_idle_notify_interval = 1 mins > mail_max_userip_connections = 500 > mail_plugins = " stats quota imap_quota imap_stats" > } > protocol pop3 { > mail_max_userip_connections = 500 > mail_plugins = " stats quota" > pop3_client_workarounds = outlook-no-nuls oe-ns-eoh > } >
Re: Too many references: cannot splice
Hi, I've been running the patched version for the last few days, and I can confirm that I have not seen any errors. I still have to check it with 4.x kernels, but it's working as expected with the current Debian Stable one. Regards, Luis ugalde. On Wed, Oct 26, 2016 at 12:59 PM, Timo Sirainen <t...@iki.fi> wrote: > On 26 Oct 2016, at 11:14, Luis Ugalde <forondar...@gmail.com> wrote: > > > > Hi, > > > > Could you please have a look at https://lkml.org/lkml/2016/2/2/538 and > see > > if this makes any sense to you? I've been checking kernel changes > > between linux_3.16.7 and linux_3.16.36, and this has popped out. Could > this > > be the reason for the "too many references" errors? > > Does the attached patch help? > > > > > > > Regards, > > > > Luis Ugalde. > > > > On Thu, Oct 13, 2016 at 3:47 PM, Luis Ugalde <forondar...@gmail.com> > wrote: > > > >> Hi, > >> > >> > >> A while ago I sent an email regarding these "*ETOOMANYREFS* Too many > >> references: cannot splice." that we've seen since Debian updated the > Jessie > >> kernel to > >> > >> 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > >> > >> while older kernels, like 3.16.0-4-amd64 #1 SMP Debian > >> 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. > >> > >> I was wondering if no one else is getting these errors, or if you know > any > >> workarounds that might probe useful, apart from downgrading the kernel. > >> > >> > >> I would say that the infrastructure we're running is quite standard, > with > >> directors balancing users to NFS backed dovecot servers. > >> > >> > >> Best regards, > >> > >> Luis Ugalde. > >> > >> > >> > >> > > >
Re: Too many references: cannot splice
Hi, Could you please have a look at https://lkml.org/lkml/2016/2/2/538 and see if this makes any sense to you? I've been checking kernel changes between linux_3.16.7 and linux_3.16.36, and this has popped out. Could this be the reason for the "too many references" errors? Regards, Luis Ugalde. On Thu, Oct 13, 2016 at 3:47 PM, Luis Ugalde <forondar...@gmail.com> wrote: > Hi, > > > A while ago I sent an email regarding these "*ETOOMANYREFS* Too many > references: cannot splice." that we've seen since Debian updated the Jessie > kernel to > > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > > while older kernels, like 3.16.0-4-amd64 #1 SMP Debian > 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. > > I was wondering if no one else is getting these errors, or if you know any > workarounds that might probe useful, apart from downgrading the kernel. > > > I would say that the infrastructure we're running is quite standard, with > directors balancing users to NFS backed dovecot servers. > > > Best regards, > > Luis Ugalde. > > > >
Too many references: cannot splice
Hi, A while ago I sent an email regarding these "*ETOOMANYREFS* Too many references: cannot splice." that we've seen since Debian updated the Jessie kernel to 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 while older kernels, like 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 showed no errors at all. I was wondering if no one else is getting these errors, or if you know any workarounds that might probe useful, apart from downgrading the kernel. I would say that the infrastructure we're running is quite standard, with directors balancing users to NFS backed dovecot servers. Best regards, Luis Ugalde.
Re: ETOOMANYREFS related errors
Hi, I've been running a Debian-Jessie backported 4.4 kernel for the last few days, and I can confirm the same issue happening with this kernel as well. Regards, Luis El vie., 11 mar. 2016 a las 9:01, Luis Ugalde (<forondar...@gmail.com>) escribió: > El vie., 11 mar. 2016 a las 2:08, Timo Sirainen (<t...@iki.fi>) escribió: > >> On 11 Mar 2016, at 03:48, Luis Ugalde <forondar...@gmail.com> wrote: >> > >> > Hi, >> > >> > I'm starting to see, on a pretty standard Debian Jessie installation, >> some >> > error messages that are apparently related to the ETOOMANYREFS errno. >> > >> > Firstly, the mail log shows this: >> > dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many >> references: >> > cannot splice >> >> Apparently because Linux thinks the same fd has been passed around >> recursively too many times: >> http://lkml.iu.edu/hypermail/linux/kernel/1101.0/01917.html >> >> But Dovecot doesn't pass it recursively. It's only passed once from >> pop3-login to pop3 process. >> >> > Is this something that Dovecot should be able to handle, or is it >> strictly >> > Debian/libc/MySillyMistake related? >> > >> > #uname -a >> > Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 >> > (2016-01-17) x86_64 GNU/Linux >> >> I wonder if there's a new kernel change that started detecting the >> recursion wrong. >> >> > Yes, It's started to happen with the latest kernels from the Stable > branch. This is the latest I have installed: > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 > > Older Debian kernels are not showing up anything. This one, for example: > 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 > > > Regards. >
Re: ETOOMANYREFS related errors
El vie., 11 mar. 2016 a las 2:08, Timo Sirainen (<t...@iki.fi>) escribió: > On 11 Mar 2016, at 03:48, Luis Ugalde <forondar...@gmail.com> wrote: > > > > Hi, > > > > I'm starting to see, on a pretty standard Debian Jessie installation, > some > > error messages that are apparently related to the ETOOMANYREFS errno. > > > > Firstly, the mail log shows this: > > dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many > references: > > cannot splice > > Apparently because Linux thinks the same fd has been passed around > recursively too many times: > http://lkml.iu.edu/hypermail/linux/kernel/1101.0/01917.html > > But Dovecot doesn't pass it recursively. It's only passed once from > pop3-login to pop3 process. > > > Is this something that Dovecot should be able to handle, or is it > strictly > > Debian/libc/MySillyMistake related? > > > > #uname -a > > Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 > > (2016-01-17) x86_64 GNU/Linux > > I wonder if there's a new kernel change that started detecting the > recursion wrong. > > Yes, It's started to happen with the latest kernels from the Stable branch. This is the latest I have installed: 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 Older Debian kernels are not showing up anything. This one, for example: 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 Regards.
ETOOMANYREFS related errors
Hi, I'm starting to see, on a pretty standard Debian Jessie installation, some error messages that are apparently related to the ETOOMANYREFS errno. Firstly, the mail log shows this: dovecot: pop3-login: Error: fd_send(pop3, 18) failed: Too many references: cannot splice And then the login process fails: dovecot: pop3-login: Internal login failure (pid=34388 id=1) (internal failure, 1 successful auths): user=, method=PLAIN, rip=rip, lip=lip, session= Is this something that Dovecot should be able to handle, or is it strictly Debian/libc/MySillyMistake related? #uname -a Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux #dovecot --version 2.2.13 #dpkg -l | grep -E 'dovecot-core|libc6|linux-image' ii dovecot-core 1:2.2.13-12~deb8u1amd64 ii libc6:amd642.19-18+deb8u3amd64 ii libc6-i386 2.19-18+deb8u3amd64 ii linux-image-3.16.0-4-amd64 3.16.7-ckt20-1+deb8u3 amd64 Best Regards, Luis