Re: cyrus master fails with status 71

[ellie timoney via Info-cyrus]

On Tue, Nov 8, 2016, at 05:01 AM, Eric Cunningham wrote:
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPCNT): 
> Invalid argument
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPIDLE): 
> Invalid argument
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPINTVL): 
> Invalid argument
> 
> Why is lmptunix complaining about options passed to imapd?

There's a bug here:

Based on a comment ("/* tcp only */"), master/service.c is trying to
limit the application of the tcp_keepalive config to TCP sockets only. 
But it's doing this by checking for the type being SOCK_STREAM -- which
is not exclusive to TCP sockets.  Notably, AF_UNIX domain sockets may
also have type SOCK_STREAM, and this looks like the default
configuration for "lmtpunix".

So, those errors are because it's trying to set TCP_* socket options on
a socket that is not a TCP socket.  (Notice too that it doesn't error
when trying to set SO_KEEPALIVE, because that's a valid SOCK_STREAM
option.  It only fails to set the TCP_* options.)

Looks like this has been fixed on master already.  I'll see if the fix
backports to 2.5 cleanly, but in the meantime you can ignore these
errors.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[Bron Gondwana via Info-cyrus]

On Tue, 8 Nov 2016, at 05:01, Eric Cunningham via Info-cyrus wrote:
> Are such numbers of imapd processes to be expected?

Depends on your usage patterns of course.

> Why is lmptunix complaining about options passed to imapd?

Because the names of the config files are bogus for historical reasons.  
imapd.conf should really be called settings.conf and cyrus.conf should really 
be called master.conf.  It's definitely not a config file for just the imap 
daemon.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian search databases

[Bron Gondwana via Info-cyrus]

On Mon, 7 Nov 2016, at 23:23, Adam Tauno Williams via Info-cyrus wrote:
> On Thu, 2016-11-03 at 10:47 +1100, Bron Gondwana via Info-cyrus wrote:
> > Just out of interest - is anyone other than Fastmail currently using
> > the Cyrus Xapian-based search system?
> 
> Not using Xapian.

Cool.  It turns out I didn't have to make it all backwards incompatible just 
yet, because instead I just detect if it's using the old Chert backend and 
force a reindex of all email to upgrade to Glass.

(but can still read from a mix of both, so you can phase the upgrade)

This is going to the rest of FastMail production today after running happily 
for the past 4 days for ~5% of our users.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[Eric Cunningham via Info-cyrus]

Hi Ellie, we've been running with your patch since Oct 25 and haven't 
encountered any issues with imapd exiting, thus far.  But, now that 
imapd has had a chance to run uninterrupted for almost 2 weeks, the 
number of imapd processes/connections has steadily climbed every day. 
This morning, it was near 16,000.  This system has a total of 1400 accounts.


To try and control this growth, per 
https://cyrusimap.org/imap/faqs/o-toomanyprocesses.html I've set the 
following:


To cyrus.conf, added "-U 50" option to the SERVICES section for imapd:

  imap  cmd="imapd -U 50" listen="imap" prefork=60
  imaps cmd="imapd -s -U 50"  listen="imaps"prefork=150


To imapd.conf, added the following tcp_keepalive options:

  tcp_keepalive: 1
  tcp_keepalive_cnt: 1
  tcp_keepalive_idle: 30
  tcp_keepalive_intvl: 900

After restarting imapd, the following are now being logged repeatedly:

Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to 
setsocketopt(TCP_KEEPCNT): Invalid argument
Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to 
setsocketopt(TCP_KEEPIDLE): Invalid argument
Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to 
setsocketopt(TCP_KEEPINTVL): Invalid argument



So, a couple of questions for the list:

Are such numbers of imapd processes to be expected?
Why is lmptunix complaining about options passed to imapd?

Thank you.

-Eric


On 10/25/16 8:23 PM, ellie timoney via Info-cyrus wrote:

Hi Eric,

Patch attached.  I'd appreciate if you could advise whether this helps.
Though I guess you won't be able to tell for a couple of weeks.

If it doesn't cause any new problems (I don't expect it to), then it
will be included in 2.5.11 (whenever that comes out).

Cheers,

ellie

On Wed, Oct 26, 2016, at 10:04 AM, ellie timoney via Info-cyrus wrote:

accept failed: Software caused connection abort


Some sleuthing suggests that "Software caused connection abort"
corresponds with "ECONNABORTED".

The man page on my system for accept(2) unhelpfully defines this as:


   ECONNABORTED
  A connection has been aborted.


But some digging around online suggests that this situation occurs when
a client connects, but subsequently disconnects  (RST) before the server
gets around to accept()ing the connection.  When the server does
eventually accept(), the accept() fails with this error.

Which sounds to me like we want to treat ECONNABORTED similarly to
EAGAIN, not as a fatal OS error.  I'll have a patch up for this shortly.

Cheers,

ellie

On Wed, Oct 26, 2016, at 09:27 AM, Eric Cunningham via Info-cyrus wrote:

Having repeatedly experienced the "status 71" issue, I've been
incrementally bumping it's value up.  It's currently set to 32768 (!)
and that value was in place when it most recently failed.


On 10/25/16 4:21 PM, Shawn Bakhtiar via Info-cyrus wrote:

H.. if that’s the case could you be hitting the the maximum number
of accepts??

Check the 11.11.1.2. kern.ipc.soacceptqueue section of the FreeBSD handbook

https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html

Given the load you described perhaps 128 is just not enough?




On Oct 24, 2016, at 1:22 PM, Eric Cunningham via Info-cyrus
> wrote:



=
 Eric Cunningham
 Information Services - http://whoi-it.whoi.edu
 Woods Hole Oceanographic Institution - http://www.whoi.edu
 Woods Hole, MA  02543-1541 phone: (508) 289-2224
 fax: (508) 457-2174   e-mail: ecunning...@whoi.edu

=

On 10/24/2016 03:45 PM, Bron Gondwana via Info-cyrus wrote:

On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:

Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2
(release-p7) ports tree.

The cyrus master process is failing periodically (every 1-2 weeks) as
follows:

Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps
path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71
Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in
READY state: terminated abnormally
Oct 22 07:38:48 imap1 master[7767]: too many failures for service
imaps/ipv4, disabling until next SIGHUP

This prevents new connections by clients until cyrus is restarted.  I've
looked around the web but have not seen this issue reported.

A little background:

Our initial thought on this was that we were running out of listen
queues so have upped that incrementally from the default of 32 to a
current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l
option, with increased kern.ipc.soacceptqueue set to 32768, but that
hasn't helped.  Sometimes the "status 71" occurs during periods of light
use during off hours, like on Saturday mornings.

We have ~1400 imap accounts, though the number of impad processes hovers
around 3,000-4,000.  There have been spikes observed as high as 

Re: Xapian search databases

[Adam Tauno Williams via Info-cyrus]

On Thu, 2016-11-03 at 10:47 +1100, Bron Gondwana via Info-cyrus wrote:
> Just out of interest - is anyone other than Fastmail currently using
> the Cyrus Xapian-based search system?

Not using Xapian.

-- 
Adam Tauno Williams  GPG D95ED383
Systems Administrator, Python Developer, LPI / NCLA



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Memory unauthorized access issues in byte2search in Cyrus IMAP

[Egoitz Aurrekoetxea via Info-cyrus]

Good morning,


I have been checking the Cyrus IMAP 2.3.19 and 2.3.18 code because I 
have observed some issues in UID SORT commands in the IMAP protocol. 
When performing a command


like ". UID SORT (SIZE) US-ASCII ALL TEXT avanzada" in a mailbox where 
matches were found caused you to obtain in a debug (or non debug I 
think) log the following entry :


Oct 31 09:17:21 hostname master[78064]: process 78268 exited, signaled 
to death by 11


Lines like this are seen when a process has been signaled by the kernel 
with signal 11. Have been reading this signal is sent to a proccess when 
it performs an unauthorized memory


access attemp (an out of the own variable, pointer... etc, storage 
room). After debugging the code with GDB and doing several checks, have 
seen the issue came from the byte2search()


function when a piece of the string s->substr was trying to be stored in 
b. Concretely the third if in the loop :



for (i = 0, cur = 0; i < s->max_start; i++) {
/* no more active offsets */
if (s->starts[i] == -1)
break;

/* if we've passed one that's not ongoing, copy back */
if (cur < i) {
s->starts[cur] = s->starts[i];
}
/* check that the substring is still maching */
if (b == s->substr[s->offset - s->starts[i]]) {


The issue was caused there because s->starts[i] in this place, was not 
being able to be accesed because it was pointing to to data outside 
s->starts. After searching where this array was being initialized


and it's memory allocated (which was in search_init function), I tried 
to allocate 10 bytes more for that pointer. After doing it, there were 
no more issues. So I tried allocating just one byte more which it seemed


to be enough too (at least for the patterns I have searched for). At 
this moment I understood this pointer (s->starts which was a 
search_state->substr pointer inside the search_state structure) was not 
having


enough room for all the content needed to be stored, or at least accesed 
when calling it. I checked then the code of Cyrus 2.3.18 and 2.3.19 but 
didn't see any kind of differences in the part of the memory


allocation (in search_init()) or usage (in bytesearch) for s->starts. I 
deciced to check Cyrus 2.4 code and I saw it's room was being allocated 
the following way :



s->starts = xmalloc(s->max_start * sizeof(size_t));


instead of that in 2.3 was done :


s->starts = xmalloc(s->max_start * sizeof(int));


So I understood s->starts should be allocated to the size of a size_t 
type defined variable size, instead to the size of an integer variable n 
times. After replacing it, has seen definitively all seemed to be


working. So wouldn't Cyrus 2.3 sources have this allocation in 
search_init done with sizeof(size_t) instead of the sizeof(int)?. I 
think this is important because else, when the first character of a


pattern is repeated more than one time, the pattern has a would say 
patlen of 8-9 bytes and matches exist in the mailbox, that search would 
end up with a proccess died due to a signal 11.



My env is FreeBSD RELENG_9_0 OS with a Cyrus 2.3.18_1 port. Am I wrong, 
shouldn't that allocation be changed?.



Thanks a lot for your time,

Best regards,


--


sarenet
*Egoitz Aurrekoetxea*
Departamento de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es 
www.sarenet.es 

Antes de imprimir este correo electrónico piense si es necesario hacerlo.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus