Bug 3113 / Re: Last call for Cyrus 2.3.14

[Дилян Палаузов]

Hello,

Ken Murchison wrote:

If there are any outstanding issues that you believe still need to be
addressed in 2.3.14, please let me know ASAP.



I'm for considering bug 3113 ( 
http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3113 ) into the new 
release.


Със здраве,
Дилян

Anonymous CVS / Wiki

[Дилян Палаузов]

Hello,

How can I access the anonymous CVS for cyrus-imap?

http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/AnonymousCVS tries to 
document this, but does not.


Thanks in advance for your answer,
  Дилян

Cyrus Imap and Automake

[Дилян Палаузов]

Hello,

if I rewrite the build system of Cyrus imap 2.4(.10) to use Automake to 
generate the Makefile.in-files, will the patch be accepted in reasonable 
time in git/master?


The advantages of using Automake:
  * there is no more need to deal with problems concerning parallel builds
  * preferences in /etc/config.site will be honoured
  * with a Makefile.am it is very easy to make cyrus imap use shared 
libraries with libtool (see bug 3095, and email from 24.06.2011 on 
cyrus-devel: currently static binaries are huge). [At least from my 
perspective adding libtool support afterwards is easy]
  * Makefile.am files are easier to maintain than Makefile.in files 
(from my perspective)

  * make install, make dist, make distclean will just work (correctly)
  * the output of 'make' can be made to look nicer  (with silent-rules)

Possible problems:
  * I am not sure right now, how can 'make install' be tweaked to 
install files in /usr/cyrus/bin, but I guess this can be tweaked in the 
worst case by moving the files from /usr/bin to /usr/cyrus/bin
  * automake 1.11's lex.am file has to be patched in order to 
acknowledge, that lex can output files other than lex.c and lex. 
However this tweak is only necessary, when rebuilding Makefile.in from 
Makefile.am (=not done by end users).


I like to have one Makefile.am per project, as described in the article 
Recursive Make Considered Harmful by Peter Miller 
(http://aegis.sourceforge.net/auug97.pdf).  If it is for some reason not 
possible to have at the end a single Makefile.am for cyrus imap, then at 
least I would like to reduce the number of automake files as much as 
possible.  (With a single Makefile, when you change a file in 
cyrus-imap/lib and recompile, you are forced to recompile not only 
libimap.a, but all binaries, that depend on this library; the output 
looks nicer with silent-rules as it is not printed, that the directory 
is changed; and having a single Makefile speeds up the loading and 
proceeding it, compared to loading several Makefiles)


Any comments?

Със здраве
  Дилян

Re: Cyrus Imap and Automake

[Дилян Палаузов]

Hello Thomas,

Thanks for your feedback.   

My idea is to improve the current build system by generating the 
Makefile(s).in by automake.


I have some experience with Autotools.  I have absolutely nothing 
against CMake or other build systems, except I have no coding experience 
with them.  If you think that CMake is more suitable than Autotools, if 
somebody wants to change the build system to make use of CMake and if 
there are no objections for cyrus imapd + CMake, then I have no problems 
with CMake.


Parallelization race conditions will be avoided by having a single 
Makefile.in for all subdirectories.


Със здраве
  Дилян


Please take a look at CMake and consider it before you start investing time 
into Autotools.


That is why I posted the question before starting.


On 29.07.2011 18:14, Thomas Jarosch wrote:

Hi Дилян,

here's some feedback about your build system question.
Note: I'm not one of the cyrus core developers.


if I rewrite the build system of Cyrus imap 2.4(.10) to use Automake to
generate the Makefile.in-files, will the patch be accepted in reasonable
time in git/master?


Have you considered alternatives to GNU Autotools?

We have experience with GNU Autotools in our company projects as well as
open source projects for several years now.

We have found that it has several shortcomings:

1. Autotools version conflicts

You can compile a released source package without any Autotools on your
system. But as soon as you

a) want to develop
b) want to install a patch which modifies the build system (like a new path
to a library, something that adds a new file,...). This is often happens
as part of packaging for .rpm or .deb.

you need Autotools on your machine. If the Autotools version on your machine
and the one used to build the release are not compatible you can't build.

Installing a different Autotools version on a given distribution without
breaking something or fixing a huge list of dependency problems is nearly
impossible. I have experience with this...

2. Build speed and parallelization

The configure run can't be run on multiple cpu cores at once and is usually
slow.

Running compile jobs on multiple cores at once can become tricky if you have
internal dependencies like libs that are included in your project. Sometimes
we experienced race conditions or cases where you had to start the build
process two or three times to get a successful build.

3. Documentation and learning

There is a book about Autotools (http://sourceware.org/autobook/). I have
read it and don't think that it explains how to use and troubleshoot
Autotools in a good way. Maybe it's hard to explain because in Autotools
everything is a macro that calls a macro that calls a macro...

I think the learning curve of Autotools is very steep and finding out what's
really happening during troubleshooting is very time consuming.

Because of these shortcomings we have used CMake for new projects for about
a year. We have made good experience with it. We have now started to port
all Autotools projects over to CMake.

Please take a look at CMake and consider it before you
start investing time into Autotools.

Best regards,
Thomas

Automake / Re: Cross-compiling Cyrus in Buildroot

[Дилян Палаузов]

Hello,


- cyrus-imapd: static binaries are huge, cannot build them shared;
- cyrus-imapd: choose binaries to (build and) install;


Not solved. The good thing is that Дилян Палаузов Dilyan Palauzov
volounteered to update the cyrus-imapd buildsystem
(http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.cyrus-develsearchterm=automakemsg=3040),

which would fix these two issues, and possibly also the previous one with 
imtest.


I want just to say, that I will work on the automake-ization at the end 
of September.  My idea is first to switch completely to Automake and 
once things work, to add libtool (shared libraries) support.


Even if there were voices, that other build system might be better, as 
far as I remember, nobody stated he would like to adjust cyrus imap for 
that systems.


Moreover I initially wrote that, to the extend possible, I prefer to 
have a single Automake file for all sub-directories.  There were 
responses justifying this idea is suboptimal.  I think that most of you 
have not practically compared building a source tree with single 
Makefile versus one Makefile per directory, and the obstacles expressed 
are theoretical.  I suggest to make a single Makefile, and afterwards 
consider all your feedback, to see what is better.


I mean, for my projects I use this single Makefile approach, and from 
your emails I do not get what is wrong with the approach and why shall I 
not use it (for cyrus imap, for my projects, in general).


Със здраве
  Дилян

Automake and some questions

[Дилян Палаузов]

Hello,

I started working on Automake + cyrus imap and about the half of the 
work is done.  I have some questions:


* lib/crc32.c is included in both lib/libcyrus.a und lib/libcyrus_min.a, 
but the four crc32_ functions are only used within imap/libimap.a (in 
imap/message.c and map/mailbox.c).  Is it fine, if lib/crc32.c is 
removed from both lib/libcyrus.a and lib/libcyrus_min.a and added to 
imap/libimap.a ?


* lib/signals.h and imap/signals.h are identical and lib/signals.c and 
imap/signals.c are similar.  Would it be wise to remove one signals.h 
and one signals.c? (And who wants to unify both signals.c?)


* sieve/Makefile.dist generates sieve/sieve.c with $(YACC) $(YFLAGS) -p 
addr addr.y . Automake allows to mention addr.y as source and it 
generates the addr.c and addr.h files automatically (without writing 
anything extra).  But I cannot pass the -p addr to addr.y in Automake. 
 Is it fine to add in addr.y '%name-prefix addr' which is the same as 
passing -p addr on the command line? (This works with bison, but I do 
not know if it works with other yacc derivatives)


* What is Smakefile used for?

* I rewrite some '#include file.h' to '#include lib/file.h', if 
file.h is in the lib directory.  I believe it makes clearer for the 
developers which files are meant and requires less CPPFLAGS=-I... and 
.  Objections?


Moreover I think makedepend is not necessary now and have removed it in 
my copy.


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

syslog/, lib/gai.h, man/rmnews.8

[Дилян Палаузов]

Hello,

make install as of cyrus imap 2.4.11 installs in /usr/include/cyrus/ all 
lib/*.h files, except auth_pts.h, chartable.h, exitcodes.h, wildmat.h 
and gai.h .


gai.h is only used within configure.in .

Shall the files auth_pts.h, chartable.h, exitcodes.h and wildmat.h be 
installed in /usr/include/cyrus/ ?

Shall the file gai.h be removed from lib/ and integrated in configure.in ?


Does somebody use the implementation in the syslog/ directory, and can 
you compile it with gcc?  When I try to compile the included syslog in 
cyrus-imap2.4 I get the errors:


For syslog/syslog.c

In file included from syslog.c:66:0:
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.1/include/varargs.h:4:2: 
error: #error GCC no longer implements varargs.h.
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.1/include/varargs.h:5:2: 
error: #error Revise your code to use stdarg.h.


For syslogd.c:
syslogd.c: In function ‘wait3’:
syslogd.c:204:16: error: argument ‘status’ doesn’t match prototype 
/usr/include/sys/wait.h:169:16: error: prototype declaration
syslogd.c:982:18: error: conflicting types for ‘sys_errlist’ 
/usr/include/bits/sys_errlist.h:28:30: note: previous declaration of 
‘sys_errlist’ was here


syslog/logger.c is included in the tarball, but it is nowhere used.

Can the syslog directory be removed?


Cyrus-imap does not install executable named rmnews.  Can the files 
man/rmnews.8 and doc/man/rmnews.8.html be removed?


Greetings,
  Дилян
attachment: dilyan_palauzov.vcf

Double sieveshell and installsieve

[Дилян Палаузов]

Hello,

in the cyrus-imapd repository, in both cyrus-imapd-2.4 and master, the 
sieveshell and installsieve are presented twice:


sieveshell is in /SIEVE/scripts/sieveshell and 
/perl/sieve/scripts/sieveshell.pl


installsieve is in /installsieve/installsieve and in 
perl/sieve/scripts/installsieve.pl,


and installsieve/libinstallsieve.a and perl/sieve/lib/libisieve.a 
contain more or less the same files.


My question is, what is the intended difference between the two 
sieveshell-s, between the two installsieve-s and between 
libinstallsieve.a and libisieve.a ? In case there is not supposed to be 
any difference, can the one of the copies be removed?


Със здраве
  Дилян

Re: Automake Support for cyrus-imapd 2.5

[Дилян Палаузов]

Hello,


commit imap/Makefile.in: add .c.snmp: recipe

The old makefile will rebuild pushstats.c if the snmpgen script changes.
  The new one won't.


Please feel entitled to fix it.


commit lib/mkchartable.pl move output from stderr to stdout

In mkchartable.pl, you need to handle the open() failing.

Also, mkchartable.pl can fail partway through, e.g. if one of the
charset/*.t files is not readable.  In the old makefile, such a failure
would trigger this code

-|| (rm -f chartable.c  exit 1)

which would break the build and avoid half-creating a chartable.c so
that the next attempt to build was also (correctly) fail.  In the new
makefile there is no such safeguard: a partly finished chartable.c would
be created and the next attempt to build would (incorrectly) not fail at
this point.


Please feel entitled to fix it.


commit Prepend the path in #include to .et -  .h files

I am deeply confused about why this is necessary, can you explain?

Getting rid of these is obviously useful:

-#include ../lib/imapurl.h
-#include ../lib/util.h

but I don't see why you would change

-#include imap_err.h
+#include imap/imap_err.h

without changing any of the other #include lines around it.

Also, here

-CPPFLAGS = -I.. -I$(srcdir)/../sieve -I$(srcdir)/../imap
-I$(srcdir)/../lib @COM_ERR_CPPFLAGS@ @CPPFLAGS@ @SASLFLAGS@
+CPPFLAGS = -I.. -I$(srcdir)/../lib @COM_ERR_CPPFLAGS@ @CPPFLAGS@
@SASLFLAGS@

it might be a good idea to use paths down from $(top_srcdir) instead of
up-and-around from $(srcdir), like this

CPPFLAGS = -I$(top_srcdir) -I$(top_srcdir)/lib @COM_ERR_CPPFLAGS@
@CPPFLAGS@ @SASLFLAGS@

Generally, I think it would be a good thing to have a clear consistent
and documented policy for how we #include headers.  We don't really have
that now, and I don't think this commit gets us any closer.


The point is, that some source files are always in $(srcdir), while 
others might be in $(srcdir) or in $(builddir).  Let's say if the files 
are not in $(srcdir), they are generated in $(builddir).  These files 
are prefixed with the path, as in #include imap/imap_err.h, so that 
the build works (this was my conclusion, when I was trying to build with 
VPATH).


The #include policy, which I have followed was: CPPFLAGS = 
$(top_srcdir), $(top_builddir), $(top_srcdir)/lib and 
$(top_builddir)/lib + external dependencies.


The reason to include both $(top_srcdir) and $(top_builddir) is that 
some headers might be in srcdir, but others might be generated in 
$(builddir).


The reason to include lib/ is that those files get at the end installed 
under /usr/include/cyrus . My logic is, the files which depend on files 
in installed in /usr/include/cyrus, need CPPFLAGS=-I  pointing to the 
place where the headers files are.  This might be $(srcdir)/lib, but it 
might be /usr/cyrus/include .



commit Move def. Makefile.in:(PACKAGE/VERSION) to configure.ac:AC_INIT

Looks good, but here

+AC_INIT([cyrus-imapd], [2.5],
[http://bugzilla.cyrusimap.org],,[http://www.cyrusimap.org])

it might be nice to have the version on it's own line, as we know that
the version string will be modified once for every release.


You have to put the version anyway on some line.  Currently it is on the 
AC_INIT line, you can move it to another line, but I do not see reasons 
to choose a line different from AC_INIT.



commit lib/Makefile.in: Reorder LIBCYR(M)_(OBJS, HDRS) alphabetically

Looks good.  While you're doing that, how about splitting them up one
per line?  We have all sorts of headaches doing git rebases of commits
which add or remove .c files because of this.


If you check the final Makefile.am, you will see that nearly everything 
is ordered alphabetically.  First are the files from the imap directory, 
then the files from imtest, lib, man, master, netnews, notify... Within 
each directory, the targets are also orderted: first idled, them imap, 
ipurge, lmtpd.  For each target, the sources files are also ordered 
alphabetically.


This makes it very easy to find something, whatever you are looking for.

With the unordered LIBCYR(M)_OBJS it was very hard to find that some 
files are included in both LIBCYRM_OBJS and LIBCYR_OBJS, which does not 
make sense, since all targets linking with libcyrus.a link also with 
libcyrus_min.a .


Okay, we can split them one per line, if you want.  But you can also put 
them all on the same line.  Having them all on the same line, it is 
quite easy to find where the change is.



commit lib/Makefile.in: Reorder LIBCYR(M)_(OBJS, HDRS) alphabetically

+   $(srcdir)/brearch.h

s/brearch/bsearch/

Please fix it.



+LIBCYRM_OBJS = assert.o hash.o imapopts.o libconfig.o mappedfile.o
mpool.o \
+   retry.o signals.o strhash.o util.o xmalloc.o xstrlcat.o
xstrlcpy.o \
+   @IPV6_OBJS@ map_@WITH_MAP@.o

map_@WITH_MAP@.o  is out of alphabetical order.


Yes, but I put the @..@ files temporary at the end.  In the final 
Makefile.am the order is alphabetical.



commit lib/Makefile.in: 

Re: Automake Support for cyrus-imapd 2.5

[Дилян Палаузов]

Hello,

My comments on Greg's 2nd comments on my comments on my commits:


commit */Makefile.in: add top_(builddir,srcdir) to CPPFLAGS

+CPPFLAGS = -I$(top_srcdir) -I$(top_srcdir)/lib -I$(top_builddir)
-I$(top_builddir)/lib @COM_ERR_CPPFLAGS@ @CPPFLAGS@ @SASLFLAGS@

Surely the correct order is

-I$(top_builddir)
-I$(top_srcdir)
-I$(top_builddir)/lib
-I$(top_srcdir)/lib

?


Yes.



Also, why do we have -I. -I$(srcdir) in some CPPFLAGS but not others?


The final Makefile.am does not have -I$(srcdir), probably as 
intermediate step -I$(srcdir) was necessary.



commit perl/Makefile*: enable building perl modules out of srcdir

+   if [ -f ../${srcdir}/$$d/Makefile.PL -a ! -f
Makefile ]; then \

This breaks if $srcdir is an absolute rather than a relative path.  In
fact so does this

+  srcdir=../${srcdir}/$$d \
+  top_srcdir=../$(top_srcdir) \

It might be a whole lot simpler to generate Makefile.PL from
Makefile.PL.in at configure time.  Then, you could run the Makefile.PL
from configure using AC_CONFIG_COMMANDS.

+INSTALLDIRS =  'vendor',

Please don't do that!  I just had to do a whole bunch of futzing to
avoid it :(


INSTALLDIRS =  'vendor' was initially there, then it was removed by you 
on Friday with commit abd6dcd95abd329b3a (Revert Bug 3652 - install 
Perl modules into 'vendor' dirs), but somehow was not merged very 
correctly in my branch.


I think the perl builds deserve a separate discussion.


commit imap/Makefile.in: order everything alphabetically

My comments from lib/Makefile.in about splitting these up 1 per line
also apply here.

In LOBJS, imapparse,o, message.o, saslclient.o, saslserver.o are out of
order.

In IMAPDOBJS, proxy.o is out of order.

In PROGS, cyr_dbtool and cyrdump are out of order

There's a reason why $SERVICE ought to be linked first, or at the very
least before any libraries from outside the project: it contains main()
and must have the first definition of main() seen by the linker even if
some library also defines main().


Yes, but in the final Makefile.am things are correct.

In master:Makefile is written: SUBDIRS = @SERVER_SUBDIRS, and 
SERVER_SUBDIRS is master imap according to master:configure.in.


master:master/Makefile.in contains
LOBJS = service.o service-thread.o
all: $(LOBJS)

so I thought the reason for the ordering of master/ before imap/ was to 
make sure, that service.o is compiled in master/, before changing to imap/.


As with linking, in Automake there are _SOURCES and _LDADD.  $SERVICE 
goes into _SOURCES, any libraries go into _LDADD.  It is not really 
possible with Automake to force linking a library before linking $SERVICE.


Please feel entitled to correct the order of cyr_df and cyrdump.


cyr_info links masterconf.o but does not depend on it.


cyr_info does depend on masterconf.o .

commit rename $service_path to $servicedir


+AC_DEFINE_UNQUOTED(SERVICE_PATH,$servicedir,[Directory to use for
service binaries])

Don't you want to rename SERVICE_PATH too?


Yes, this could be also done.


Also, the difference between $servicedir and $bindir is slight; in the
default arrangement one will be /usr/cyrus/bin and the other /usr/bin.
I would hope that eventually we could eliminate $cyrus_prefix and
replace $servicedir with $bindir.


Yes, but this is not really related to the switch to Automake.


  commit remove cyrus_prefix from every Makefile.in, as it is unused

Since three days ago, it is used, in the perl/ directories to expand
$PERL_PREINSTALL.  Otherwise, looks good.


Yes, please fix it somehow.


commit Initial Automake support

Do we need the 'depend' target?  Surely automake handles that for us.

+   SIEVE_LIBS=../sieve/libsieve.a

$(top_builddir)


The depend target is removed in subsequent commits together with makedepend.

SIEVE_LIBS can be extended with $(top_builddir).  In fact at the end, 
SIEVE_LIBS and SIEVE_OBJS are used only for Cunit checks.  So if 
Cunit/Makefile.in can be merged to /Makefile.am SIEVE_LIBS and 
SIEVE_OBJS can be completely removed from configure.ac .


Със здраве
  Дилян

Re: Automake Support for cyrus-imapd 2.5

[Дилян Палаузов]

Hello,


Third and final tranche of review comments.



commit Makefile.am: merge com_err/et/Makefile.in

I think this has already been covered, but using top_builddir in the
Makefile.in rather than in configure.ac will break when the library is
actually found in /usr/lib

+LIBS = ${top_builddir}/@COM_ERR_LIBS@


It is now fixed.

imap/imap_err.h imap/imap_err.c: imap/imap_err.et
cd imap  $(COMPILE_ET) ../$(top_srcdir)/imap/imap_err.et

and the same for sieve/sieve_err.et, imap/mupdate_err.et and 
imap/nntp_err.et



commit Makefile.am: merge lib/Makefile.in

+AM_CFLAGS = @PERL_CCCDLFLAGS@ $(EXTRACFLAGS)

This is wrong.  PERL_CCCDLFLAGS should only be used for the code which
is built into the perl extension shared library.  In particular, it
contains -fPIC which we do not want anywhere else if we can avoid it.


Greg, I admit that you have more understanding about Perl than me.  I 
kindly ask you to fix the Perl stuff, including the 
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3678 issue.


Why are those EXTRACFLAGS necessary at all?  Can't AM_CFLAGS be used 
instead?



+#this is from lib/test/Makefile.in, however testglob.c does not exist
in
+#/lib/test, instead testglob2.c is there.
+#lib_test_testglob_LIRBARIES = lib/libcyrus.a lib/libcyrus_min.a
-ldb-4.0

Everything in lib/test/ will be either moved to cunit/ or removed,
eventually, and is currently not built. You probably shouldn't bother
with it.


Yes, but in order for make dist to put the all files in the tarball, 
these files need to mentioned somehow.  Anyway, once the 
automake-building works, I suggest to have a separate discussion on how 
to implement make snapshot/make dist/generation of xversion.h, etc. 
 Let's stick now to the dependancy and compilation issues.



commit Makefile.am: merge sieve/Makefile.in

+   sieve/test/testExtension 

Not only is this line far too long, but the directory is sieve/tests/
not sieve/test/


I renamed all sieve/test/ to sieve/tests/ .


commit Makefile.am: merge notifyd/Makefile.in

Looks good, although I note that notifytest is not built at all
currently but is built in the new Makefile.


notifytest is build by make check.


commit Makefile.am: merge syslog/Makefile.in

I think it's time we deleted the entire syslog/ directory.  I don't
believe there's any platform on which it will be built, nor be useful.


I deleted the syslog/ files and the SYSLOG Automake conditional.

Please check if the syslog checks in configure.ac can be deleted, too.


commit Makefile.am: merge ptclient/Makefile.in

+#ptclient_ptextract_SOURCES = imap/cli_fatal.o imap/mutex_fake.o
ptclient/ptextrac.c

s/ptextrac/ptextract/

+ptclient_ptloader_SOURCES = ptclient/ptloader.c ptclient/ptloader.h
ptclient/afskrb.oc ptclient/ldap.c imap/mutex_fake.o
master/service-thread.c

s/afskrb.oc/afskrb.c/

Fixed.


commit Makefile.am: merge installsieve/Makefile.in


It is unclear why this directory exist, since there is no place, that
installs installsieve.


Another relic, let's remove it too.


I deleted installsieve/* .


commit Include SIEVE/scripts/sieveshell.pl in make dist


 It is unclear why this script is neede (and installsieve/installsieve)
in parallel to perl/sieve/scripts/(sieveshell|installsieve).pl


I don't see any good reason for this to continue existing.


I deleted it.


commit Makefile.am: merge doc/(Makefile.dist, text/Makefile)

+man/cyradm.1.html:
+   @$(MKDIR_P) man
+   pod2html $(top_srcdir)/perl/imap/cyradm.sh  man/cyradm.1.html
+   rm man/pod2htm*


man/cyradm.1.html should depend on $(top_srcdir)/perl/imap/cyradm.sh


The dependency on cyradm.sh is fixed now.


Also, why the rm??


master:doc/Makefile.in contains:

dist:
mkdir -p man
for man in ../man/*.[1-9]; \
do \
echo === $$man ===; \
groff -man -Thtml $$man  man/`basename $$man`.html; \
done

pod2html ../perl/imap/cyradm.sh  man/cyradm.1.html
pod2man ../perl/sieve/scripts/sieveshell.pl  ../man/sieveshell.1
pod2html ../perl/sieve/scripts/sieveshell.pl  
man/sieveshell.1.html


rm -f groff-html-*.png pod2htm*

fig2dev -L png murder.fig murder.png
fig2dev -L png netnews.fig netnews.png

(cd text; make)

And the rm comes from there.


+   @$(MKDIR_P) doc
+   @$(MKDIR_P) doc/text

The first of these lines is redundant.


Fixed.


commit Makefile.am: merge imap/Makefile.in

+ $(LN_S) -f lmtpd lmtpprodyx

s/lmtpprodyx/lmtpproxy/

Fixed.


+   cd imap  $(top_builddir)/../$(COMPILE_ET)
../$(top_srcdir)/imap/imap_err.et

This is wrong in so many ways.

See above.


+AM_CONDITIONAL([WITH_OPENSSL], [test $with_openssl != no])

For consistency, the name of the conditional should be OPENSSL

Fixed.

Със здраве
  Дилян

Re: Automake Support for cyrus-imapd 2.5

[Дилян Палаузов]

Hello Greg,

On 18.04.2012 06:31, Greg Banks wrote:

On Tue, Apr 17, 2012, at 03:09 PM, Dilyan Palauzov wrote:

commit Prepend the path in #include to .et -   .h files
[...] I don't see why you would change

-#include imap_err.h
+#include imap/imap_err.h

without changing any of the other #include lines around it.


I prefer to keep (AM_)CPPFLAGS minimalistic and have everthing
relative to the top_dirs.  Automake includes automatically -I.
-I$(srcdir) and -I(the directory with config.h = top_builddir), so it
is not necessary to add those explicitly (even if it is done for
top?builddir).


Yes, I agree so far.


Specifying the path to generated and included *.h
files is the minimal effort required to keep AM_CPPFLAGS as small as
possible (top_dir and top_dir/lib).


I have two problems with this statement.

Firstly, I don't see why generated .h files need to be treated any
differently.  If they're mentioned in BUILT_SOURCES then they get built
before everything else in the 'all' target, so by the time any other .c
file is compiled or the dependencies are extracted the generated .h
already exists and is found correctly.


Because generated files go in $(builddir), and the others are in $(srcdir).


Secondly, lib/ is not the only directory from which .h files are
included.  I did a quick count based on gcc-generated .dep files and
found

3578 headers included into .c files, comprising
1456 included from .
174 include from top_dir into files in other directories (e.g. config.h)
1872 included from top_dir/lib into files in other directories
61 included from top_dir/imap into files in other directories
14 included from top_dir/sieve into files in other directories
1 included from top_dir/master into files in other directories


Yes.  I have prefixed those files also with the path.

If you want to put path-prefixes to all included *h files, and nobody 
has objections, you can add the path-prefixes and insert in 
AM_INIT_AUTOMAKE([...]) nostdinc.



commit lib/Makefile.in: Reorder LIBCYR(M)_(OBJS, HDRS) alphabetically

Looks good.  While you're doing that, how about splitting them up one
per line?  We have all sorts of headaches doing git rebases of commits
which add or remove .c files because of this.


Okay, we can split them one per line, if you want.  But you can also put
them all on the same line.  Having them all on the same line, it is
quite easy to find where the change is.


Ah, but *finding* is not the problem; my text editor has a working
search function.

The problem is when you're rebasing a commit that adds a single source
file, and the upstream changes you're rebasing around add or remove
another source file.  With multiple source files per line, you are far
more likely to get spurious context damage which prevents the git merge
from succeeding, which then requires a manual merge conflict fix, which
is annoying slow and error-prone.  When Bron and I were actively working
on the conversations branch we would suffer through dozens of these
spurious merge conflicts per week.  See commit
3aa2792f402a27687cf0ecdcd38654716436ec0c


The problem would probably not have appeared, if all the files were on
a single, very long line.


On the contrary, such an arrangement is the worst possible case because
every change to the list of files is a merge conflict waiting to happen.
  It would have resulted in conflicts every single time we had to rebase
around a commit which added a source file, rather than about half of
them.


If you want to put each SOURCE file on a separate line in Makefile.am , 
you are free to do so.


Със здраве
  Дилян

Re: In preparation of Cyrus IMAP 2.5: autoconf and automake

[Дилян Палаузов]

Hello,


To this end, we encourage you to clone the GIT repository master branch
and attempt a build, or, alternatively, download the following snapshot
release:

http://git.cyrusimap.org/cyrus-imapd/snapshot/cyrus-imapd-2.5-snapshot-autoconf-and-automake.tar.gz

The canonical build process we think applies, generally speaking, is:

$ autoreconf -v


The idea at the end is to create the tarballs with make dist, which 
includes Makefile.in, config.h.in and configure, so that autoreconf 
does not have to be run by the ones downloading the tarball (and it does 
not matter, if git manages those files).


Apart from including the hand-written files, and the files listed by 
automake --help, some more generated files are going to be included by 
make dist, but I am not sure yet which files exactly.



lib/auth_krb5.c fails to build, because autofoo does absolutely nothing to find 
krb5.h:
lib/auth_krb5.c, line 60: cannot find include file: krb5.h


I guess this has not working also in cyrus-imapd-2.4 .


Undefined   first referenced
symbol in file
krb5_free_principal lib/libcyrus.a(auth_krb5.o)
krb5_realm_compare  lib/libcyrus.a(auth_krb5.o)
krb5_build_principallib/libcyrus.a(auth_krb5.o)
uuid_unparse_lower  imap/libimap.a(mailbox.o)
krb5_get_default_realm  lib/libcyrus.a(auth_krb5.o)
krb5_parse_name lib/libcyrus.a(auth_krb5.o)
krb5_init_context   lib/libcyrus.a(auth_krb5.o)
krb5_free_context   lib/libcyrus.a(auth_krb5.o)
krb5_unparse_name   lib/libcyrus.a(auth_krb5.o)

So not only is autofoo not bothering to find the header file, it isn't adding 
the krb5 libs, either! How on earth does this link for anyone?! Manually adding 
LIBS='-lkrb5' to configure fixes all but the uuid problem.


The problem with -lkrb5 might disappear if you upgrade automake.  On my 
system, automake adds -lkrb5 to LIBS and uses $LIBS when linking (if 
libkrb5 is found during configure).  Which file exactly can't you link?



imap/append.c, line 431: syntax error before or at: 1

using sun as a variable isn't the best choice of variable names

Adding '-Xc' to the sun studio compiler to keep it from polluting the namespace would fix that, but 
would also undefine __FUNCTION__ causing the build to fail on the snmp bits. So I renamed all 
instances of sun to mysun in imap/append.c and imap/idlemsg.c.


I have renamed sun to mysun in imap/(append,idlemsg).c and uploaded 
the changes.


Със здраве
  Дилян

Prefer pod2html over groff to create sieveshell.1.html from sieveshell.pl / sieveshell.1

[Дилян Палаузов]

Hello,

the file doc/man/sieveshell.1.html is created with

doc/man/sieveshell.1.html: $(top_srcdir)/perl/sieve/scripts/sieveshell.pl
pod2html $(top_srcdir)/perl/sieve/scripts/sieveshell.pl  $@

man/sieveshell.1 is created with
pod2man $(top_srcdir)/perl/sieve/scripts/sieveshell.pl  $@

and all other doc/man/ files are created with

  for man in $(top_builddir)/man/*.[1-9] $(top_srcdir)/man/*.[1-9]; \
do \
   groff -man -Thtml $$man  doc/man/`basename $$man`.html; \
done
  rm -f doc/groff-html-*.png

Is there any reason to prefer
pod2html $(top_srcdir)/perl/sieve/scripts/sieveshell.pl  
doc/man/sieveshell.1.html

over
groff -man -Thtml $(top_srcdir)/man/sieveshell.1  
doc/man/sieveshell.1.html

and can the special rule for sieveshell.1.html be deleted?

The same question applies to (doc/)man/cyradm.1(.html), with the remark, 
that man/cyradm.1 is not created, only doc/man/cyradm.1.html . (Shall it 
stay this way?)


Със здраве
  Дилян

Re: Add VACATION :seconds support

[Дилян Палаузов]

Hello,

To patch 1: autobuild: Fix directory handling
  Why do you use autobuild.sh instead of autoreconf -vi?

To patch 2,3,4: Change vacation day to seconds
  I would suggest adding a new opcode to the sieve bytecode for 
vacation :seconds, that is used for all uploads in the future, in a 
way similar to the implementation in bc_eval.c of B_FILEINTO_ORIG and 
B_REDIRECT_ORIG.  This will allow current scripts to continue running 
without the need to be recompiled.  For future scripts, always the 
seconds are stored, even if :days was provided.  Currently bc_eval 
supports all versions starting from BYTECODE_MIN_VERSION = 0x03.


I mean, in addition to your patches,
  - replacing in bytecode.h B_VACATION with B_VACATION_ORIG on line 
130, and adding a new B_VACATION at the end of enum bytecode (could be 
called also B_VACATION and B_VACATION_SECONDS)
  - fixing bc_dump and sieved to be able to dump both B_VACATION and 
B_VACATION_ORIG commands
  - fixing bc_eval to be able to execute both B_VACATION and 
B_VACATION_ORIG commands.


Със здраве
  Дилян


On 08.05.2012 11:54, Philipp Hahn wrote:

Hello,
some time ago I implemented RFC 6131http://tools.ietf.org/html/rfc6131,
which adds a variante of the VACATION extension supporting seconds instead of
days. The original patch was for version 2.2.x and implemented a draft version.

There are some issues I would like to get a feedback for:

autobuild.sh fails for me on my local system, because my directory layout
doesn't seem to match your build environment. I took the liberty to fix some
issues to make it usable for me. The first patch is actually independent from
the rest.

The change is relatively easy iff the value used in compiled SIEVE scripts is
changed from days to seconds. As far as I understood the logic, this requires a
version bump in the sieve byte code, which I did in the second patch. Since
duplicate_mark() already uses seconds since the UNIX epoch, no change is
required there.
This reduces the maximum span for vacation messages from 2^31 to 68 years, but
I think even that should be enough.

:seconds 0 is explicitly allowed, which would send a vacation mail for every
message. (Actually this was the motivation for implementing this extension) In
other contexts this might be undesired, but I found no easy way to make
min_response and max_response configurable. Because of that lowering
min_response is done in the last patch.

There currently is no explicit check for 0= seconds= 2**31.

For converting days to seconds and the inverse I always explicitly wrote 24 *
60 * 60. Is there a good location for defining this as a macro?

Philipp Hahn (4):
   autobuild: Fix directory handling
   Change vacation day to seconds
   Add VACATION :seconds support
   Support VACACTION :seconds 0

  autobuild.sh   |   18 ++-
  imap/lmtp_sieve.c  |6 ++--
  sieve/README   |3 ++
  sieve/bc_dump.c|2 +-
  sieve/bc_emit.c|4 +-
  sieve/bc_generate.c|4 +-
  sieve/bytecode.h   |3 +-
  sieve/interp.c |8 +++--
  sieve/message.c|4 +-
  sieve/message.h|2 +-
  sieve/script.c |   11 +++
  sieve/script.h |1 +
  sieve/sieve-lex.l  |1 +
  sieve/sieve.y  |   31 ---
  sieve/sieve_interface.h|6 ++--
  sieve/sieved.c |2 +-
  sieve/test.c   |5 ++-
  .../actionExtensions/uberExtensionActionScript.key |   10 ++
  .../actionExtensions/uberExtensionActionScript.s   |   13 +++-
  sieve/tree.h   |2 +-
  20 files changed, 94 insertions(+), 42 deletions(-)

Re: Add VACATION :seconds support

[Дилян Палаузов]

Post scriptum:

Also sieve/interp.c:sieve_listextensions(sieve_interp_t *i) has to be 
fixed, so that it includes vacation-seconds in its return value.  The 
result of sieve_listextensions, including vacation-seconds, is printed 
by timsieved's CAPABILITY command.


Extending sieve_listextensions() probably requires adding 
vacation-seconds in sieve_extensions in lib/imapoptions.




On 08.05.2012 13:32, Дилян Палаузов wrote:

Hello,

To patch 1: autobuild: Fix directory handling
Why do you use autobuild.sh instead of autoreconf -vi?

To patch 2,3,4: Change vacation day to seconds
I would suggest adding a new opcode to the sieve bytecode for vacation
:seconds, that is used for all uploads in the future, in a way similar
to the implementation in bc_eval.c of B_FILEINTO_ORIG and
B_REDIRECT_ORIG. This will allow current scripts to continue running
without the need to be recompiled. For future scripts, always the
seconds are stored, even if :days was provided. Currently bc_eval
supports all versions starting from BYTECODE_MIN_VERSION = 0x03.

I mean, in addition to your patches,
- replacing in bytecode.h B_VACATION with B_VACATION_ORIG on line 130,
and adding a new B_VACATION at the end of enum bytecode (could be called
also B_VACATION and B_VACATION_SECONDS)
- fixing bc_dump and sieved to be able to dump both B_VACATION and
B_VACATION_ORIG commands
- fixing bc_eval to be able to execute both B_VACATION and
B_VACATION_ORIG commands.

Със здраве
Дилян

Libtool and Support for Shared Libraries

[Дилян Палаузов]

Hello,

I wanted to upload a new branch for testing libtool/shared libraries 
functionality, but something went wrong and the 13 commits were uploaded 
on top of 'master' instead of creating a separate branch.  I am sorry 
for the surprises this might cause.  Anyway...


I have updated cyrus-imapd to use libtool and build shared (and static) 
libraries, and the binaries to use the shared libraries.  I hope you 
like the results.


Some issues:
  -- where shall the libraries be installed?  Currently libcyrus and 
libcyrus_min go under $libdir (e.g. /usr/lib), as this was before with 
using libtool


  -- libimap and libsieve are installed under pkglibdir = 
$cyrus_prefix/lib, e.g. /usr/cyrus/lib .  If the libraries are moved to 
$libdir, then they shall probably be renamed to libcyrus_imap and 
libcyrus_sieve to avoid conflicts with other libraries with the same 
(common) name


  -- I have problems with MakeMaker, it would be very, very nice if 
somebody considers to change the build system for the perl files, e.g. 
switch to something more modern and developed like Module::Build (to 
what I read MakeMaker is not developed any more).  In any case, the perl 
modules are linked with the static libraries of libcyrus_min and 
libcyrus, so currently for the binaries the shared libraries are used, 
for the perl-stuff the static libraries are used and cyrus-imapd 
compiles by default both shared and static libraries.


  -- in the future library versioning could be used, e.g. to associate 
library-version to a cyrus-imapd release or so, at the moment my idea 
was just to make shared libraries and these are un-versioned.


Enjoy!
  Dilian
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries

[Дилян Палаузов]

Hello,


While we're at it, I'd love to split things like 'mailbox.c' which are
really libraries into a separate directory from 'imapd.c' which is a
system daemon only run by master from cyrus.conf and separate again
from cyr_dbtool.c which is a tool designed to be mainly run by humans.


I would suggest, that
the sources of libcyrus are moved to src/lib/cyrus,
the sources of libcyrus_min are moved to src/lib/cyrus_min,
the sources of libimap are moved to src/lib/imap,
the sources of libsieve are moved to src/lib/sieve,
the sources of the services from imap/ are moved to src/service,
probably the installable header files are moved to src/include,

and then check what to do with the remaining files.

With services I mean fud, imapd, lmtpd, pop3d, smmapd and sync_server.

Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries

[Дилян Палаузов]

Hello,

I put the libraries initially under $cyrus_prefix/lib (/usr/cyrus/lib), 
because the directory $cyrus_prefix already existed.


If the intention is to move the files from $cyrus_prefix / $servicepath 
(/usr/cyrus/bin), to /usr/libexec and /usr/bin, then the shared 
libraries can be renamed to libcyrus-imap and libcyrus-sieve and 
installed under $libdir.



commit configure.ac, Makefile.am: Add libtool support

So, with no arguments to LT_INIT() are we getting both shared and static

versions of the libraries by default?

Yes.  But when the package is compiled, this can be further tuned with 
passing options to ./configure (--disable-shared, --disable-static).



commit convert com_err/et/libcom_err.a to a libtool archive

You should also fix the 2nd argument to db_panic() in lib/cyrusdb_berkeley.c.


I am not sure: gcc does not complain.  I think it matters only when 
errno is a global variable, in db_panic() it is local.



Also, I notice that libcom_err.la is 'noinst'.  How do you expect that cyrus 
executable will be able to find this code at runtime on a platform which has 
shared libraries but no system com_err library?


When the compilation of the internal com_err is requested, the files 
from libcom_err.la are compiled with -fPIC and statically linked in 
libimap and libsieve.  That is how the files are found at run time. 
This has the disadvantage, that running timsieved and lmtpd with shared 
libraries support, loads libimap and libsieve, so the statically linked 
code of libcim_err is loaded twice.  An option would be to install 
com_err.la as shared libcyrus_com_err, which is loaded by both libimap 
and libsieve.


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries

[Дилян Палаузов]

Hello,


Also, I notice that libcom_err.la is 'noinst'.  How do you expect that cyrus 
executable will be able to find this code at runtime on a platform which has 
shared libraries but no system com_err library?


When the compilation of the internal com_err is requested, the files
from libcom_err.la are compiled with -fPIC and statically linked in
libimap and libsieve.


Two questions -

1) is this happening by accident because we're currently adding -fPIC to
the global CFLAGS via @PERL_CCLDFLAGS@ ?


According to my understanding, -fPIC is added, so that the compiled .o
files, can be inserted in a shared library and this is independent from
@PERL_CCLDFLAGS@.


Ah.  I was under the impression that libtool+automake would build two
different .o files, one with -fPIC for shared libraries and one without
for static libraries and executables.


For installable libraries, this is true, because both static and shared 
libraries are installed.  For noinst_ (convenience) libraries, this is 
also true (but I do not know why libtool builds .o files twice and I do 
not know, if the files that are supposed to be built without -fPIC are 
build in fact with @PERL_CCLDFLAGS@).


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Libtool and Support for Shared Libraries (2)

[Дилян Палаузов]

Hello,

I have integrated the comments about libtool/shared libraries and at the 
end the changes are:


  -- in libcyrus_min:libconfig:config_read one parameter is added for 
config_need_data, so that this variable does not have to be global and 
presented in programs, not using config_read, update all invocations of 
the function to use the new API
  -- in libimap:global:cyrus_init add one parameter (config_need_data), 
so that its value can be passed to config_read; update all invocations 
of cyrus_init;
  -- create by default only shared libraries, this can be further tuned 
with ./configure --enable-static / --disable-shared
  -- rename libsieve and libimap to libcyrus_sieve and libcyrus_sieve 
and install them in $(libdir) (e.g. /usr/lib)
  -- make the perl/{imap,sieve/managesieve} modules use the shared 
libraries libcyrys_min and libcyrus.  This might lead to a problem, when 
the tests in perl/imap/t/ are run, before the shared libraries are 
installed, as the tests depend on the shared libraries, but the tests 
are not instructed to where to find thw libraries.
  -- when there is no system-wide libcom_err, compile the bundled one 
as libcyrus_com_err and eventually install it in $(libdir)


I hope with uploading now the separate branch dev/libtool on 
git.cyrusimapd.org I will not mess up the things again.  The problem the 
last time (Build failed in Jenkins: cyrus-imapd-master #571) was caused 
by unclean aclocal.m4:



+ autoreconf -i -f -v -Icmulocal
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I cmulocal --force -I cmulocal
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --include=cmulocal --force
autoreconf: running: /usr/bin/autoheader --include=cmulocal --force
autoreconf: running: automake --add-missing --copy --force-missing
Makefile.am:69: library used but `RANLIB' is undefined
Makefile.am:69:   The usual way to define `RANLIB' is to add `AC_PROG_RANLIB'
Makefile.am:69:   to `configure.ac' and run `autoconf' again.
Makefile.am:73: Libtool library used but `LIBTOOL' is undefined
Makefile.am:73:   The usual way to define `LIBTOOL' is to add `LT_INIT'
Makefile.am:73:   to `configure.ac' and run `aclocal' and `autoconf' again.
Makefile.am:73:   If `LT_INIT' is in `configure.ac', make sure
Makefile.am:73:   its definition is in aclocal's search path.
autoreconf: automake failed with exit status: 1
+ fatal 'Can'\''t run autoreconf'
+ echo 'imapd/tools/jenkins-build.sh: Can'\''t run autoreconf'
imapd/tools/jenkins-build.sh: Can't run autoreconf
+ exit 1
Build step 'Execute shell' marked build as failure
Skipping Cobertura coverage report as build was not SUCCESS or better ...
Recording test results


On my system I do not get autoreconf: configure.ac: not using Libtool 
and I guess running git clean will clean the cause of the error message.


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries (2)

[Дилян Палаузов]

Hello,


commit Makefile.am: move the defintion of COM_ERR_LIB here

I don't think there's any point having COM_ERR_LIB and COM_ERR_LIBS,
just one should do, but we can tweak that later.


libimap and libsieve have to depend on libcom_err.a whenever the bundled 
libcom_err.a is used, so that the bundled libcom_err.a is build before 
libimap and libsieve are linked.  Any ideas how to achieve this, that 
are different from what I did?



commit convert lib/libcyrus_min.a to libtool archive
commit convert lib/libcyrus.a to libtool archive

Looks good, except that the Perl build is broken.  I get this:

make[2]: Entering directory `/home/gnb/software/cyrus/imapd/perl/imap'
make[2]: *** No rule to make target `-lcyrus', needed by `subdirs'.
Stop.
make[2]: *** Waiting for unfinished jobsinfo

I've been wrestling with this most of the day but libtool + MakeMaker =
one enormous headache :(


I moved -lcyrus -lcyrus_min in Makefile.PL.in from MYEXTLIB to 
OTHERLDFLAGS, so that the target subdirs does not depend anymore on 
-lcyrus -lcyrus_min.  Please try with the uploaded fix.  (I can 
understand why you got this error, but I do not get it.)



Also, is there a way of getting rid of these warnings:

   /bin/bash ./libtool   --mode=install /usr/bin/install -c imtest/imtest
   '/home/gnb/software/cyrus/inst/usr/cyrus/bin'
libtool: install: warning: `lib/libcyrus.la' has not been installed in
`/usr/cyrus/lib'
libtool: install: warning: `lib/libcyrus_min.la' has not been installed
in `/usr/cyrus/lib'

because there's quite a lot of them and they're confusing the script I
use to find compile time warnings.


Pass --prefix to ./configure .  The resulting binaries have to be aware 
where the shared libraries are (when the shared libraries are not on 
standard locations) and probably using make install DESTDIR= with 
libtool is either wrong or implemented/handled wrong in Automake/libtool.


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Libtool and Support for Shared Libraries (3)

[Дилян Палаузов]

Hello,

to sum up the issues with Libtool:

- the perl shared objects IMAP.so and managesieve.so do not contain 
Library rpath on Greg's computer, but have it on mine.  I have not 
checked the details, but I think on my computer -rpath comes from the 
fact, that to build LDFLAGS the file libdb-4.8.la is considered (when I 
configure with BDB support), the file contains libdir=/usr/lib64 and 
this leads to -Wl,-rpath,/usr/lib64 in LDFLAGS.  At the same time, 
Automake includes automatically _rpath = -rpath $(libdir) variables 
(at least with --enable-sieve for libsieve, and with --enable-server for 
libimap, unconditionally for libcyrus_min and libcyrus, and when the 
bundled libcom_err is used, it is also linked with -rpath $(libdir)). 
However the cyrus-libraries are installed in $(libdir), and not 
/usr/cyrus/lib, because we want them to be on a standard location, which 
is considered by the dynamic loader, and are in ldconfig's path.  So 
what is the problem, when the perl shared objects do not contain -rpath? 
 I think the other executables shall also not contain it.


- make install DISTDIR= causes warnings from libtool, that state, that 
the libraries are not installed on the place they are intended to be 
installed (as configured) and the executables are not going to work (as 
they will not be able to find the libraries under the DESTDIR= root). 
This warning gives right information, on the other side Greg says, that 
the make install DISTDIR= is a normal process that shall not lead to 
warnings.  I asked at libtool-...@gnu.org , but I if they do not answer, 
I can't say anything more about this.


Apart from the suboptimal building of the perl parl, which was 
suboptimal before, and is even more suboptimal now, I think that are all 
points with libtool/shared libraries.


As the shared libraries were not discussed, is somebody against using 
shared libraries with cyrus-imapd 2.5 / merging the dev/libtool branch?


Със здраве
  Дилян

On 30.05.2012 12:26, Jeroen van Meeuwen (Kolab Systems) wrote:

On 2012-05-29 1:25, Greg Banks wrote:

On Mon, May 28, 2012, at 09:33 PM, Dilyan Palauzov wrote:

 [...]probably using make install DESTDIR= with
 libtool is either wrong or implemented/handled wrong in
Automake/libtool.

 Well, that sucks.

Why do not you use ./configure --prefix=$(DESTDIR), so that make
install DESTDIR=somewhere is not necessary? To my understanding
installing in DESTDIR is used to create packages,


So we now generate dozens of warnings when doing a straightforward,
entirely normal, and unavoidable step in the packaging process? I don't
see how that's acceptable.



While of course in my realm of packaging, I could use ./configure, what
I actually use is %configure. It expands to a predefined set of standard
configure options such as --prefix=/usr, --libdir=/usr/lib64, etc, along
with first exporting a bunch of variables.

When the make install DESTDIR=/some/where is issued, we point it to the
root directory of a buildroot (%{buildroot}) and expect the other
./configure options to kick in so that everything is still finally
placed in the correct directories (i.e.%{buildroot}etc/ for
--sysconfdir=%{_sysconfdir}, %{buildroot}usr/lib64 for
--libdir=%{_libdir}, etc. exec dir, libexec dir, ...) for which, if the
prefix were to be set to the buildroot root directory, we would need to
add the options for all the other --*dir= configure options as well.

Kind regards,

Jeroen van Meeuwen

attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries (3)

[Дилян Палаузов]

Hello,

concerning the rpath for the perl shared objects IMAP.so and 
managesieve.so, after all I found that MakeMaker uses as linker by 
default the value returned by $Config(ld) (visible by running perl -V 
under Linker and Libraries), which is sometimes ld='cc' and sometimes 
ld='ld'.


If I do not overwrite the default LD-variable (= $config(ld)), then it 
might be cc, which could be gcc or not.  I have no idea how to pass 
linker flags, if the linker is cc, but not gcc (with gcc the 
parameters are passed with -Wl,param).  So on the one side, I do not 
know how to instruct $config(ld)/the linker in a portable way by passing 
-rpath parameters to include rpath in the .so file.


On the other side, at least gnu ld, considers the variable LD_RUN_PATH, 
when there are no -rpath parameters passed to it and included the value 
of LD_RUN_PATH as rpath in the generated shared object.  This is how I 
understand the documentation of -rpath in gnu ld:


Options:
[...]
-rpath=DIR ... If `-rpath' is not used when linking an ELF executable, 
the contents of the environment variable `LD_RUN_PATH' will be used if 
it is defined.


MakeMaker does set LD_RUN_PATH, before invoking the linker.  This leads 
to including rpath=$(libdir) in perl/imap/IMAP.so and 
perl/sieve/managesieve/managesieve.so .  At least this is the logic on 
my system.  If on other systems the rpath is not included in IMAP.so or 
managesieve.so, and LD_RUN_PATH in the resulting perl/imap/Makefile is 
set up properly, I cannot say where the problem is.


The discussion with RPATH is only about the perl shared objects.  The 
other executables and shared libraries have correct RPATH.


With DISTDIR I meant DESTDIR.

About the libtool warnings that come when using DESTDIR, they might be 
suppressed by passing --no-verbose or --quiet to libtool, however it 
might cause other useful warning to disappear and I guess it cannot be 
passed to libtool only in install-mode, but has to be in the compile 
mode, too (if it is managed by Automake).


Със здраве
  Дилян
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries (3)

[Дилян Палаузов]

Hello,

then I suggest to build both static and shared libraries, to link all 
services and user programmes with the shared libraries, to link the 
IMAP.so and manageiseve.so files with the static libraries, and resolve 
by this way the need to insert RPATH in IMAP.so and managesieve.so .


I guess libtool has to be run for every step, and this makes the things 
very messy.


Running libtool with --quiet during make install DESTDIR= (achieved by 
make install DESTDIR=... LIBTOOLOPTIONS=--quiet) steal leads to 
warnings that the shared libraries are not installed on their final 
destination.


Със здраве
  Дилян

On 04.06.2012 04:13, Greg Banks wrote:



Sent from my iPhone

On 03/06/2012, at 20:50, Дилян Палаузов dilyan.palau...@aegee.org wrote:


Hello,

concerning the rpath for the perl shared objects IMAP.so and
managesieve.so, after all I found that MakeMaker uses as linker by
default the value returned by $Config(ld) (visible by running perl -V
under Linker and Libraries), which is sometimes ld='cc' and
sometimes ld='ld'.

If I do not overwrite the default LD-variable (= $config(ld)), then
it might be cc, which could be gcc or not. I have no idea how to
pass linker flags, if the linker is cc, but not gcc (with gcc the
parameters are passed with -Wl,param). So on the one side, I do not
know how to instruct $config(ld)/the linker in a portable way by
passing -rpath parameters to include rpath in the .so file.



This is very platform specific, which is why libtool exists.

Perhaps you could arrange for MakeMaker to run libtool for the link
step? This might result in needing to run libtool for *every* step,
which may be awkward.

Relying on LD_RUN_PATH is just as platform specific as using -rpath.

Greg.
attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries (3)

[Дилян Палаузов]

Hello,

now libcyrus and libcyrus_min are compiled once as shared libraries, 
once under perl/ as non-static convenience libraries,
  perl/imap/IMAP.so and perl/sieve/managesieve/managesieve.so link 
statically with the non-static convenience libraries,
  so that IMAP.so and managesieve.so do need neither libcyrus and 
libcyrus_min at run time nor the RPATH with those libraries at tun time.


I hope this is the end with the shared libraries, before they can be merged.


I guess libtool has to be run for every step, and this makes the things
very messy.


Maybe not. According to the libtool documentation, you can convince
libtool --mode=link to build a native .a file using the obvious
options -o foo.a.  So you might be able to convince the main
Makefile.am to build a native .a and link against that in the MakeMaker
makefile.


This was done, when for all libraries, both shared and static versions 
were build and perl used just the static versions of libcyrus_min and 
libcyrus to link with IMAP.so and managesieve.so and libimap and 
libsieve were compiled and installed statically, just because.  Anyway, 
I hope now with using only libcyrus and libcyrus_min as non-static 
convenience libraries to have solved the problem.


After merging the support for sieve-seconds, jenkins did not rebuild 
sieve/sieve.c from sieve/sieve.y (it considered probably sieve.c newer 
as sieve.y), I fixed tools/jenkins-build.sh to run make 
maintainer-clean (instead of make distclean) before each run, so that 
sieve/sieve.c is deleted.  This caused deleting also 
imap/rfc822_headers.[ch], which were not rebuild, as ./configure was not 
run with --enable-maintainer-mode.  If gperf is installed on 
jenkins.cyrusimapd.org the next build shall run normally.


Със здраве
  Дилян

On 05.06.2012 09:02, Greg Banks wrote:



On Mon, Jun 4, 2012, at 04:22 PM, Дилян Палаузов wrote:

Hello,

then I suggest to build both static and shared libraries, to link all
services and user programmes with the shared libraries, to link the
IMAP.so and manageiseve.so files with the static libraries, and resolve
by this way the need to insert RPATH in IMAP.so and managesieve.so .


Using noinst_LTLIBRARIES to create a non-static convenience library
(in libtool's terminology) whose only purpose is to get linked into
IMAP.so and managesieve.so ?  This might work.


I guess libtool has to be run for every step, and this makes the things
very messy.


Maybe not. According to the libtool documentation, you can convince
libtool --mode=link to build a native .a file using the obvious
options -o foo.a.  So you might be able to convince the main
Makefile.am to build a native .a and link against that in the MakeMaker
makefile.


Running libtool with --quiet during make install DESTDIR= (achieved by
make install DESTDIR=... LIBTOOLOPTIONS=--quiet) steal leads to
warnings that the shared libraries are not installed on their final
destination.


Ah, libtool... :(

 From the documention it looks like using -R $(libdir) instead of -rpath
$(libdir) might be useful.

attachment: dilyan_palauzov.vcf

Re: Build failed in Jenkins: cyrus-imapd-master #652

[Дилян Палаузов]

Hello,

I merged today the dev/libtool branch in master.

I have no idea, why autoreconf does not run cleanly on ci.cyrusimap.org 
and complains about missing RANLIB and LT_INIT.  On my system there are 
no problems, libtool recommended to remove AC_PROG_RANLIB from 
configure.ac and LT_INIT is there.


Can somebody with local access check why autoreconf does not run?  A 
manual deletion of aclocal.m4 or autom4te.cache might solve the problem 
(or not).


Със здраве
  Дилян


+ make maintainer-clean
CDPATH=${ZSH_VERSION+.}:  cd .  /bin/sh 
http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/imapd/missing --run aclocal-1.11 -I cmulocal
  cd .  /bin/sh 
http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/imapd/missing --run automake-1.11 
--foreign
Makefile.am:70: library used but `RANLIB' is undefined
Makefile.am:70:   The usual way to define `RANLIB' is to add `AC_PROG_RANLIB'
Makefile.am:70:   to `configure.ac' and run `autoconf' again.
Makefile.am:71: Libtool library used but `LIBTOOL' is undefined
Makefile.am:71:   The usual way to define `LIBTOOL' is to add `LT_INIT'
Makefile.am:71:   to `configure.ac' and run `aclocal' and `autoconf' again.
Makefile.am:71:   If `LT_INIT' is in `configure.ac', make sure
Makefile.am:71:   its definition is in aclocal's search path.
make: *** [Makefile.in] Error 1
+ autoreconf -i -f -v -Icmulocal
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I cmulocal --force -I cmulocal
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --include=cmulocal --force
autoreconf: running: /usr/bin/autoheader --include=cmulocal --force
autoreconf: running: automake --add-missing --copy --force-missing
Makefile.am:70: library used but `RANLIB' is undefined
Makefile.am:70:   The usual way to define `RANLIB' is to add `AC_PROG_RANLIB'
Makefile.am:70:   to `configure.ac' and run `autoconf' again.
Makefile.am:71: Libtool library used but `LIBTOOL' is undefined
Makefile.am:71:   The usual way to define `LIBTOOL' is to add `LT_INIT'
Makefile.am:71:   to `configure.ac' and run `aclocal' and `autoconf' again.
Makefile.am:71:   If `LT_INIT' is in `configure.ac', make sure
Makefile.am:71:   its definition is in aclocal's search path.
autoreconf: automake failed with exit status: 1
+ fatal 'Can'\''t run autoreconf'
+ echo 'imapd/tools/jenkins-build.sh: Can'\''t run autoreconf'
imapd/tools/jenkins-build.sh: Can't run autoreconf
+ exit 1
Build step 'Execute shell' marked build as failure
Skipping Cobertura coverage report as build was not SUCCESS or better ...
Recording test results



attachment: dilyan_palauzov.vcf

Re: Libtool and Support for Shared Libraries (3)

[Дилян Палаузов]

Hello,

On 27.06.2012 22:43, Carson Gaspar wrote:

On 6/27/2012 7:36 AM, Dilyan Palauzov wrote:


Moreover, does it make any difference, if imapd links with
libcyrus_sieve and libcyrus_sieve links with libcom_err, or if imapd
links explicitly with both libcyrus_sieve and libcom_err ?


What is calling com_err? The objects in libcyrus_sieve or the objects in
imapd? If libcyrus_sieve has unresolved symbols against libcom_err,
some linkers will require additional options to allow it to link (GNU ld
and Solaris ld both default to allowing unresolved symbols when linking
shared objects, but I'm not sure about other platforms - see -z defs).


com_err, or rather the function error_message is called both by the 
objects in libcyrus_imap (as in imap/message.c) and directly by the 
objects in programs (imap/sync_client.c).



In general, you _really_ don't want a shared object depending on a
static object, although it's possible to make it work by always linking
in the static object in the final executable on many (most?) platforms.


Yes, but this is what is going to happen: if somebody requests during 
./configure to use the system-wide (static) library, it has to be used, 
and the only way to do it, is to link the programs with libcom_err.a and 
not the libcyrus_ libraries.


By the way, why does ./configure offer to link with static or dynamic 
cyrus-sasl, and by static and dynamic krb?  It is nearly the same  as 
with com_err .


Със здраве
  Дилян


attachment: dilyan_palauzov.vcf

minimal required version of (optional) tools

[Дилян Палаузов]

Hello,

is it possible to define in doc/install-compile.html the minimum 
versions of the tools, a developer needs in order to re-generate all 
source files need for the compilation of cyrus-imapd?


The needed tools are autoconf, automake, bison, (f)lex, gperf, libtool 
and possibly compile_et .


with the version requirements
  autoconf = 2.67
  automake = 1.10
  libtool = 2.2.6

What about the minimum required version of (f)lex, bison and gperf?

gperf is checked for, only when ./configure --maintainer-mode .  If 
gperf is not found, ./configure aborts (I think).  However, ./configure 
always checks for lex and bison, and does not fail, if they are not 
found (I think).


Can we stick to flex instead of lex?  In any case, flex 2.5.35 supports 
%option reentrant noyyget_FUNCTIO and noyyset_FUNCTION, which options 
are not supported by lex on ci.cyrusimap.org .  noyyget_ and noyyset_ 
permit to exclude some unneeded functions from sieve/addr-lex.c and 
sieve/sieve-lex.c, which results smaller .c files and smaller tables of 
exported symbols (which in theory means less time for the dynamic linker 
to find an exported symbol).


Със здраве
  Дилян


attachment: dilyan_palauzov.vcf

Re: minimal required version of (optional) tools

[Дилян Палаузов]

Hello,


Speaking of which...Dilyan, what is the status of the dist: target?


The only problem I am aware of is that make distclean does not clean 
the files generated by perl, and make cannot create /xversion.h, as 
there is no v2.5 tag and the distance v2.5..HEAD cannot be measured (and 
included in /xversion.h).


Със здраве
  Дилян

On 02.07.2012 11:33, Greg Banks wrote:



On Mon, Jul 2, 2012, at 11:15 AM, Bron Gondwana wrote:



And the makefiles we ship with release tarballs better bloody work
everywhere or the panda will be sad.


Good point.  Once we get the dist: target sorted out in the new build
system, we should set up a Jenkins job which builds a dist tarball from
git and then builds from the tarball.

Speaking of which...Dilyan, what is the status of the dist: target?



attachment: dilyan_palauzov.vcf

Re: minimal required version of (optional) tools

[Дилян Палаузов]

Hello,

to my opinion there are two things:

1. What do people cloning from git need in order to build the package
2. What tools versions do developers need, when they change a e.g. lex file.

About people cloning from git, if we want to make their life easier, we 
can version-manage the generated files (e.g. sieve/addr-lex.c, 
Makefile.in) and that's all - no tools are really necessary.


About requirements for developers changing the Makefile.am or flex 
files, or even the autoconf input... Well about autoconf, if somebody is 
able to make changes in autoconf input (configure.ac and cmulocal/), 
then she shall be able to locally install the latest version of autoconf 
(under ~/bin or /usr/local/bin), so the highest version of autoconf 
could be required.  About automake 1.12 and flex 2.5.35, provided that 
we are talking not about everybody who clones over git, but for those 
modifying sieve/addr-lex.l, sieve/sieve-lex.l or Makefile.am files, 
these people are soo few, that it is not necessary to talk about the 
Long Term Support releases.


Със здраве
  Дилян

On 02.07.2012 11:09, Greg Banks wrote:



On Mon, Jul 2, 2012, at 09:53 AM, Дилян Палаузов wrote:

Hello,


1) if we have any doubts about the version of tools needed to generate
parts of the source, then ship the generated source in the dist tarball
and hide the make rules in maintainer mode.  The code should be
buildable from a dist tarball on the widest possible range of platforms.


I am talking about maintainer mode.  The distributed tarballs contain
anyway the generated files and as long as the input files (e.g.
sieve/addr-lex.l are not changed, the Makefile-rules are not invoked and
it does not matter, if the tool is available or not.


Great, we're all good then :)


2) For maintainer mode, the ideal situation is that versions of tools
available in the current Long Term Support releases of common target
platforms should work without difficulty.


Which versions of autoconf, automake, bison, (f)lex, gperf and libtool
are in the current Long Term Support releases of common target platforms?


That's where *you* get to do the research.

Me, I'm lazy.  I usually start by setting the minimum version to the
version on my desktop and then decrease it any time I see a problem I
can't work around or a machine where I can't upgrade the tools.But
then I don't usually go and rewrite an entire build system of a
non-trivial project, not after the great CML2 disaster of 2001 :)


Why for maintainer mode it is not possible to stick to versions, that
are not yet in the Long Term Support releases?


Because maintainer mode is really anyone who changes the source code in
a .y file mode rather than the mode for people with write access to
the Cyrus git repo.   I don't think it's good policy in an open source
project to put an artificial speed hump in the way of users contributing
patches (compilable, tested patches).  Given that we're a server
project, the users I'm thinking of are mail system admins.  A
significant portion of those folks want stable dependable supported
platforms, which is what Long Term Support releases are all about.
FastMail doesn't operate like that, but lots of people who could be
contributing do.

In short, it's wonderful to have the latest flashy tool feature, but if
only two people in the world can build from git then we've got a
problem!

Remember, these are just my suggestions for rules of thumb, I'm not the
final arbiter.



attachment: dilyan_palauzov.vcf

linking with libtool and

[Дилян Палаузов]

Hello,

I reverted the changes causing the warnings with static in the header files.

The linking problem, Bron mentioned on 12.07., was is caused by 
(./libtool --config)'s link_all_deplibs=no .


Does anybody have an idea, when is link_all_deplibs set to no, and when 
to yes/unknown?


Със здраве
  Дилян

On 12.07.2012 00:48, Bron Gondwana wrote:



On Thu, Jul 12, 2012, at 12:17 AM, Дилян Палаузов wrote:

Hello,

can you please send the last lines of make before the crash, your
versions of automake and libtool, and the content of libcyrus_imap.la ?
   My output of make is attached.  From it can be seen, on the lines for
cyr_synclog, that libtool is invoked to link with libcyrus_imap.la, and
it expands to libcyrus_imap.so, libcyrus.so and libcyrus_min.so .


Trying it without the -j:

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I. -I./lib -I. -I./lib 
-DHAVE_CONFIG_H -I/usr/include -fPIC -g -fPIC -W -Wall -Wextra -MT 
lib/hashu64.lo -MD -MP -MF lib/.deps/hashu64.Tpo -c lib/hashu64.c  -fPIC -DPIC 
-o lib/.libs/hashu64.o
/bin/bash ./libtool --tag=CC   --mode=link gcc -fPIC  -g -fPIC -W -Wall -Wextra 
  -o perl/libcyrus_min.la  lib/assert.lo   lib/hash.lo lib/imapopts.lo 
lib/libconfig.lo lib/lock_fcntl.lo  lib/map_shared.lo   lib/mpool.lo 
lib/retry.lo lib/strarray.lo lib/strhash.lo lib/util.lo lib/xmalloc.lo 
lib/xstrlcat.lo lib/xstrlcpy.lo lib/hashu64.lo  -ldb-4.8 -lpcre -lpcreposix -lz
libtool: link: ar cru perl/.libs/libcyrus_min.a lib/.libs/assert.o 
lib/.libs/hash.o lib/.libs/imapopts.o lib/.libs/libconfig.o 
lib/.libs/lock_fcntl.o lib/.libs/map_shared.o lib/.libs/mpool.o 
lib/.libs/retry.o lib/.libs/strarray.o lib/.libs/strhash.o lib/.libs/util.o 
lib/.libs/xmalloc.o lib/.libs/xstrlcat.o lib/.libs/xstrlcpy.o 
lib/.libs/hashu64.o
libtool: link: ranlib perl/.libs/libcyrus_min.a
libtool: link: ( cd perl/.libs  rm -f libcyrus_min.la  ln -s ../libcyrus_min.la 
libcyrus_min.la )
gcc -DHAVE_CONFIG_H -I.   -I. -I./lib -I. -I./lib -DHAVE_CONFIG_H 
-I/usr/include -fPIC  -fvisibility=hidden -g -fPIC -W -Wall -Wextra -MT 
imtest/imtest_imtest-imtest.o -MD -MP -MF imtest/.deps/imtest_imtest-imtest.Tpo 
-c -o imtest/imtest_imtest-imtest.o `test -f 'imtest/imtest.c' || echo 
'./'`imtest/imtest.c
./lib/prot.h:156:12: warning: ‘prot_flush_internal’ declared ‘static’ but never 
defined [-Wunused-function]
./lib/prot.h:237:12: warning: ‘prot_setreadcallback’ declared ‘static’ but 
never defined [-Wunused-function]
./lib/prot.h:281:26: warning: ‘protgroup_copy’ declared ‘static’ but never 
defined [-Wunused-function]
mv -f imtest/.deps/imtest_imtest-imtest.Tpo imtest/.deps/imtest_imtest-imtest.Po
/bin/bash ./libtool --tag=CC   --mode=link gcc -fPIC  -fvisibility=hidden -g 
-fPIC -W -Wall -Wextra   -o imtest/imtest imtest/imtest_imtest-imtest.o 
lib/libcyrus.la lib/libcyrus_min.la -ldb-4.8 -lpcre -lpcreposix -lz -ldb-4.8 
-lpcre -lpcreposix -lz
libtool: link: gcc -fPIC -fvisibility=hidden -g -fPIC -W -Wall -Wextra -o 
imtest/.libs/imtest imtest/imtest_imtest-imtest.o  lib/.libs/libcyrus.so 
lib/.libs/libcyrus_min.so -ldb-4.8 -lpcre -lpcreposix -lz -Wl,-rpath 
-Wl,/usr/cyrus/lib
depbase=`echo imap/arbitron.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.   -I. -I./lib -I. -I./lib -DHAVE_CONFIG_H 
-I/usr/include -fPIC  -g -fPIC -W -Wall -Wextra -MT imap/arbitron.o -MD -MP -MF 
$depbase.Tpo -c -o imap/arbitron.o imap/arbitron.c \
mv -f $depbase.Tpo $depbase.Po
depbase=`echo imap/cli_fatal.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.   -I. -I./lib -I. -I./lib -DHAVE_CONFIG_H 
-I/usr/include -fPIC  -g -fPIC -W -Wall -Wextra -MT imap/cli_fatal.o -MD -MP -MF 
$depbase.Tpo -c -o imap/cli_fatal.o imap/cli_fatal.c \
mv -f $depbase.Tpo $depbase.Po
depbase=`echo imap/mutex_fake.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I.   -I. -I./lib -I. -I./lib -DHAVE_CONFIG_H 
-I/usr/include -fPIC  -g -fPIC -W -Wall -Wextra -MT imap/mutex_fake.o -MD -MP -MF 
$depbase.Tpo -c -o imap/mutex_fake.o imap/mutex_fake.c \
mv -f $depbase.Tpo $depbase.Po
/bin/bash ./libtool --tag=CC   --mode=link gcc -fPIC  -g -fPIC -W -Wall -Wextra 
  -o imap/arbitron imap/arbitron.o imap/cli_fatal.o imap/mutex_fake.o 
imap/libcyrus_imap.la -ldb-4.8 -lpcre -lpcreposix -lz -ldb-4.8 -lpcre 
-lpcreposix -lz
libtool: link: gcc -fPIC -g -fPIC -W -Wall -Wextra -o imap/.libs/arbitron 
imap/arbitron.o imap/cli_fatal.o imap/mutex_fake.o  imap/.libs/libcyrus_imap.so 
-ldb-4.8 -lpcre -lpcreposix -lz -Wl,-rpath -Wl,/usr/cyrus/lib
/usr/bin/ld: imap/arbitron.o: undefined reference to symbol 'xstrdup'
/usr/bin/ld: note: 'xstrdup' is defined in DSO /usr/cyrus/lib/libcyrus_min.so.0 
so try adding it to the linker command line
/usr/cyrus/lib/libcyrus_min.so.0: could not read symbols: Invalid operation
collect2: ld returned 1 exit status
make[2]: *** [imap/arbitron] Error 1
make[2]: Leaving directory `/home/brong/src/cyrus-imapd'
make[1]: *** [all-recursive] Error 1
make[1

Re: -fvisibility=hidden

[Дилян Палаузов]

Hello,

On 19.07.2012 03:11, Greg Banks wrote:

On Wed, Jul 18, 2012, at 10:46 PM, Дилян Палаузов wrote:

On 16.07.2012 02:08, Greg Banks wrote:


So, remind me again what actual value we're getting from this 
-fvisibility=hidden stuff again?


Initially, libcyrus_sieve had a lot of exported symbols generated by
bison and flex.  When libcyrus_sieve was loaded by the dynamic linker
the list of exported symbols (section .dynsyms) was long, so finding the
needed symbol by the dynamic linker was supposed to take long time.


Well, maybe.  On Linux at least, the linker puts a hashtable index to
the exported symbols into a section called .gnu.hash and the runtime
linker uses that to speed up searching for symbols.  So lookup is going
to be something closer to O(1) than O(N).  So I'm unconvinced by an
argument from runtime link performance.


There is always a table making the search faster than O(n), the gnu 
linker's table (--hash=style=gnu) is better than the standard table 
(--hash-table=sysv), but still, less symbols means smaller table.  The 
arguments from run time link performance are not fucking strong, but the 
resulting executable file/library are smaller.




There is an argument to be made from untidiness.

gnb@gnb-desktop 2069 nm -D --defined-only
./sieve/.libs/libcyrus_sieve.so | awk '{h[$2]++}END{for (i in h) print
h[i],i}'
3 A
10 B
3 D
115 T

That's a lot of exported symbols for such a small library :(

But you could fix that without advanced non-portable linker trickery, by
(for example) moving chunks of sieve/tree.c into sieve/sieve.y, or
#include'ing one into the other or vice versa.


The most T-symbols are generated by flex/bison and this cannot be 
tricked.  Flex can be instructed not to generate unnecessary functions 
with %option noyyset_in or alike, however flex on ci.cyrus-imapd.org 
does not support these options.  So the functions are there.


Moving functions from tree.c to sieve.y is marginal compared to the 
amount of functions created by bison/flex.


The non-portable trickery does not harm.  It is supported by GCC and 
Clang (according to 
http://clang-developers.42468.n3.nabble.com/Does-clang-support-attribute-visibility-quot-default-quot-td3944043.html 
).  Anyway, with -fvisibility=hidden approximately 60-70 symbols are 
internalized/hidden, that would be otherwise exported in the .dynsyms 
section.


Със здраве
  Дилян




Annotating in libcyrus_sieve (and the other libraries) the
functions/symbols with EXPORTED, that are needed outside the library,
and compiling with -fvisibility=hidden, keeps the list of exported
symbols short, so the dynamic linker can load the library faster.  On
the other side the resulting library is smaller.  In addition, I think
this makes things easier to maintain, as it is clear, if a function is
used outside the library (EXPORTED), only within the library (HIDDEN),
or only within the source file, it is defined (static)


I think -fvisibility=hidden is an excellent mechanism for folks
maintaining real actual libraries with defined and documented interfaces
at both the source and ABI levels, to enforce that users of their
libraries use only the supported interfaces.  Cyrus really isn't one of
those.  We have no documentation, no ABI guarantee, no versioning
mechanism.  Worse, we have a lot of truly horrible tricks which rely on
traditional linktime semantics, fatal() being the obvious example.
Those can and should be fixed, but right now they're pretty broken.


Compiling different Automake targets with different CPPFLAGS/CFLAGS
creates .o files with very long names, when the non standarf AM_CFLAGS
are used.  The reason for the long name is, that Automake assumes, that
a file can be compiled once with the non-standard (not Makefile.am-wide)
CFLAGS, and once with AM_CFLAGS, so Automake reserves its right to
create different .o files from the same .c file.  This could be avoided,
if -fvisibility=hidden is used not only for libraries but also for
executables and is put in AM_CFLAGS.


Sure, if it's one place it should be in as many places as possible.

attachment: dilyan_palauzov.vcf

Re: commit mboxevent: Rewrite JSON formatting

[Дилян Палаузов]

Hello Michel,

according to my understanding, there are no strict rules what goes in 
libcyrus, libcyrus_min and libcyrus_imap .  I am not aware of a 
guidelines, describing in which library to include what kind of file. 
While working on getting Automake/libtool in cyrus-imapd, I moved some 
files between libcyrus and libcyrus_min according to how I felt the things.


lib/parseaddr.c is used within libcyrus_sieve and imap/, so having it in 
the common library libcyrus seems logical.  I do not know, why iostat.c 
is part of libcyrus.


If nobody else expresses opinion, whether to put xjson in libcyrus or 
libcyrus_imap, it is up to you.  I just told you my opinion.


I have no opinion about the minimal required version for json.

The comment in the Makefile state, that LD_BASIC_ADD is used to link 
with programs, linking with libcyrus and libcyrus_min, while 
LD_UTILITY_ADD is used to link with all command line tools (that are not 
invoked as services from master). -lcrypto is always provided, even if 
the linked program does not use libcrypto.


Със здраве
  Дилян



On 14.08.2012 16:21, Sébastien Michel wrote:

please delete imap/Makefile.in : all the build rules are in /Makefile.am .


It's already done. Sorry, I just forgot to mention it.




As libjson supports the .pc format, you can detect libjson in
configure.ac with
PKG_CHECK_MODULES ([libjson], [json = 0.10], [check_libjson=yes],
[check_libjson=no])
and remove the configure.ac:AC_CHECK_LIB handling.  Possibly add an
AM_CONDITIONAL([LIBJSON], [$check_libjson = yes]).  I know currently
cyrus-imapd does not use the
PKG_CHECK_MODULES-routine, but on the other side there are no other
external modules, which can be detected by it.
I do not have recipe to how to trivially #define HAVE_LIBJSON when using
libjson, but this shall not be hard to handle.


I've used PKG_CHECK_MODULES and it looks fine. However I have noticed some
issues :
- By default, the macro will set up two variables, joining the given
prefix with the suffixes _CFLAGS and _LIBS.
The macro will call AC_SUBST on these variables only with pkg-config 0.24
and later
What do you prefer between always calling AC_SUBST  or testing the
pkg-config version ?

- Using PKG_CHECK_MODULES in conditional block would raise a failure.
Calling PKG_PROG_PKG_CONFIG seems to be one possible solution :
http://www.flameeyes.eu/autotools-mythbuster/pkgconfig/pkg_check_modules.html#idp1027760

- some people discourage the use of such macro :
http://stackoverflow.com/questions/10220946/pkg-check-modules-considered-harmful



I am in favour of PKG_(PROG_)PKG_CONFIG, but if you prefer not to use it,
then you can leave configure.ac as it is.


OK.
Can we consider that we support only version 0.24 minimum ?




Then shift the lib/xjson.c and lib/xjson.h to imap/,


xjson.[ch] doesn't depends on any structure/function in imap folder. As a
basic utility I think it should be located in lib.
Why do you prefer move it here ?



Provided that the json code is integrated and used within libcyrus_imap (and
not libcyrus(_min), then the files have to go under imap/ -- there are all
libcyrus_imap sources.  While for the future it would be wise, to have the
source files for libcyrus_imap in a separate directory, for the next major
release the directory structure will not change.  This has nothing to do
with being a basic utility or not.



I don't understand than you this comment from Makefile.am :
# BASIC is the libraries that every Cyrus program (except master) will
# need to link with.
#
# Note that several places in the code use -lcrypto, e.g. for SHA1 or
# MD5 algorithms, without needing SSL.  Currently we have no way of
# minimally linking such code.
LD_BASIC_ADD = lib/libcyrus.la lib/libcyrus_min.la ${LIBS}

# UTILITY is the libraries that utility programs which use Cyrus'
# mailbox and message handling code need to link with.
LD_UTILITY_ADD = imap/libcyrus_imap.la $(LD_BASIC_ADD) $(COM_ERR_LIBS)

As xjson may be linked with every Cyrus program except master (ie.
imapd, pop3d, lmtpd, cyr_expire, ipurge, ...)
I understand that it should be added in libcyrus.la or lib/libcyrus_min.la
I noticed that some files like iostat.c and parseaddr.c are not used
within lib/ but are integrated within libcyrus.la. Thus, I think I
should also integrate xjson.c within libcyrus.la (not libcyrus_min)
and not move it to imap/ ...


(in the beginning of Makefile.am, for correct CPPFLAGS) AM_CFLAGS = 
$(libjson_CFLAGS)


I don't know no what is better between that or this declaration below :
imap_libcyrus_imap_la_CFLAGS += $(JSON_CFLAGS)



Having different CFLAGS per target, implies that a single source file can be
compiled several times for different targets with different -D directives.
In order to avoid disasters, Automake renames the resulting .o files to be
very long, and to contain the target, for which the files are compiled.
This long file names are avoided, if only one (AM_)CFLAGS is used.
Currently you will not see a 

Re: Sieve with plussed delivery / 'seen flag'?

[Дилян Палаузов]

Hello,

this is similar to Bug 3695 ( 
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3695 ).


I use lmtp_fuzzy_mailbox_match: 1.

Delivery (to a plus or simple address) is proceeded first by 
imap/lmtpd.c:deliver_local() and then by deliver_mailbox().  I found 
out, that when I use an address without plus, the 7th parameter passed 
to deliver_mailbox is username (owner of the mailbox), and 
mydata-authuser (which was NULL in my tests) otherwise.


This leads later to a call of imap/append.c:append_setseen().  If 
username was passed as 7th argument to deliver_mailbox (no 
plus-address), as-internalseen is true in that function and the message 
is marked as \\Seen.  Otherwise internalseen is false and the message 
does not appear as \\Seen.  I have no investigated what as-seen_seq 
does or is supposed to do.


This patch fixes the problem on my system (addr+Sent is resolved during 
fuzzy-matching), but I don't know if it causes any side effects.


diff --git a/imap/lmtpd.c b/imap/lmtpd.c
index f70f587..5bba193 100644
--- a/imap/lmtpd.c
+++ b/imap/lmtpd.c
@@ -724,7 +724,7 @@ int deliver_local(deliver_data_t *mydata, char 
**flag, int n

/* try delivery to a fuzzy matched mailbox */
ret2 = deliver_mailbox(md-f, mydata-content, mydata-stage,
   md-size, flag, nflags,
-  mydata-authuser, mydata-authstate, 
md-id,
+  (char*) username, mydata-authstate, 
md-id,

   username, mydata-notifyheader,
   namebuf, md-date, quotaoverride, 0);
}

Comments?

Greetings
  Dilian


On 24.06.2013 16:20, Karl Pielorz wrote:


Hi,

We're running Cyrus IMAP 2.4.17. We've got sieve rules used for accounts
- and some users use 'plussed' addresses.

For example, I've setup a Sieve rule that matches on a subject of 'Test':


require [fileinto];
require [imapflags];

if header :contains [Subject] [Test] {
  addflag \\Seen;
}


This works fine e.g. if we email 'u...@example.com' - with a subject of
'Test' it'll appear in their inbox as 'Seen' mail (i.e. already read).

However, if we do a direct delivery to a subfolder e.g. email
'user+s...@example.com' - the mail appears in that folder, but appears
as 'Unseen'.

Weirdly, if we change it to do:

  addflag \\Seen
  addflag \\Flagged

And send an email to the plussed address, it arrives - is Flagged, but
has not had the 'Seen' flag added to it? (i.e. it arrives as Flagged,
Unseen).

Does anyone know why this is? - Or how we can fix it?

'anyone' has p rights to the destination folder (and during testing we
gave 'anyone' all rights to the destination) with the same effect.

Thanks,

-Karl

proxy / Re: How to prime user calendars in Cyrus 3?

[Дилян Палаузов]

Hello,

can't for this purpose the distinction between 
authorization/authentication be solely used, e.g. as in the SASL LOGIN 
mechanism?


So one provides distinct authcid and authzid, which implies the value of 
the Authorize-As header,


Greetings
  Дилян

On 05/04/2017 07:16 PM, Ken Murchison wrote:

Hi Nic,

Try adding the following header to your authenticated (as admin) requests:

Authorize-As: 


This *should* allow the admin to proxy as the specified user.  There
currently isn't any standard for proxy auth in HTTP, so I manufactured
by own header to do so.  This is how a frontend server in a Murder
proxies requests to the backend as the given user.



On 05/04/2017 12:00 PM, Nic Bernstein wrote:

Ken,
I'm back to the project of trying to migrate my own Cal/CardDav users
from DaviCal to Cyrus 3.  Problem I'm faced with is that other than
myself, my users do not have #calendars or #addressbooks in the Cyrus
mail store.  Thus my migration script fails, as destination
collections do not exist.

I do have "|caldav_create_default:| 1" set, but when pointing a
browser to "http://newjiji:8008/dav/calendars/user/" for users
who don't already have the collections created, I get a 404 error
"Mailbox does not exist."

So my question is, how does one get these users primed?  My migration
scripts use the administrative user to connect and perform the work,
so such connection is not happening as the user themselves.  If my
only recourse is to make each user connect a calendar client, then
I'll do that, but I don't think that scales well.

My concern here is dealing with sites which, like our own, have
already deployed a CalDav solution and are looking to migrate to the
inbuilt solution on Cyrus.

Any thoughts?

Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24   v. 414.272.4477
Milwaukee, Wisconsin  53213-4073


--
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

Re: proxy / Re: How to prime user calendars in Cyrus 3?

[Дилян Палаузов]

Hello,

pardon me, apparently HTTP/1.1 does not understand SASL.

Regards
  Дилян

On 05/06/2017 08:22 PM, Дилян Палаузов wrote:

Hello,

can't for this purpose the distinction between
authorization/authentication be solely used, e.g. as in the SASL LOGIN
mechanism?

So one provides distinct authcid and authzid, which implies the value of
the Authorize-As header,

Greetings
  Дилян

On 05/04/2017 07:16 PM, Ken Murchison wrote:

Hi Nic,

Try adding the following header to your authenticated (as admin)
requests:

Authorize-As: 


This *should* allow the admin to proxy as the specified user.  There
currently isn't any standard for proxy auth in HTTP, so I manufactured
by own header to do so.  This is how a frontend server in a Murder
proxies requests to the backend as the given user.



On 05/04/2017 12:00 PM, Nic Bernstein wrote:

Ken,
I'm back to the project of trying to migrate my own Cal/CardDav users
from DaviCal to Cyrus 3.  Problem I'm faced with is that other than
myself, my users do not have #calendars or #addressbooks in the Cyrus
mail store.  Thus my migration script fails, as destination
collections do not exist.

I do have "|caldav_create_default:| 1" set, but when pointing a
browser to "http://newjiji:8008/dav/calendars/user/" for users
who don't already have the collections created, I get a 404 error
"Mailbox does not exist."

So my question is, how does one get these users primed?  My migration
scripts use the administrative user to connect and perform the work,
so such connection is not happening as the user themselves.  If my
only recourse is to make each user connect a calendar client, then
I'll do that, but I don't think that scales well.

My concern here is dealing with sites which, like our own, have
already deployed a CalDav solution and are looking to migrate to the
inbuilt solution on Cyrus.

Any thoughts?

Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24   v. 414.272.4477
Milwaukee, Wisconsin  53213-4073


--
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

Re: httpd behind proxy

[Дилян Палаузов]

Hello Dave,

to what I see the Forwarded header is used by httpd only to generate correct 
URLs in the responses.

Moreover, adding for= to Forwarded: in write_forwarding_hdrs, that is never 
used, is kind of overengineering.

Greetings
  Дилан

On June 21, 2018 8:19:04 PM GMT+02:00, Dave McMurtrie  
wrote:
>On Thu, 21 Jun 2018, Дилян Палаузов wrote:
>
>> Hello,
>>
>> Nginx being proxy removes the Etag when sub(stutions) are involved 
>> (https://forum.nginx.org/read.php?2,242807,242809#msg-242809).
>>
>> If Nginx is used as proxy and it returns ETags on GET, then most 
>> probably the backend runs already on https and has the right
>hostname, 
>> so that nginx doesn't need any rewritings.
>>
>> Now, if a client sends Forwarded header and httpd, not being behind a
>
>> reverse proxy, interprets it, replacing the schema and hostname in
>the 
>> answer, e.g the URL: in /freebusy/user/... request, then the
>behaviour 
>> of httpd by interpreting the header will be correct: the client asked
>
>> for troubles and got troubles.  The troubles however do will not
>happen 
>> if httpd is behind a proxy and the proxy inserts Forwarded, as only
>the 
>> last Forwarded is supposed to be interpreted.  Irgnoring in this case
>
>> Forwarded, as this is anyway now the case, is also correct.
>>
>> So I propose removing the checks in
>imap/http_proxy.c:http_proto_host() 
>> for config_mupdate_server and proxyservers.
>
>Wouldn't that break in a murder configuration?  proxyservers is how the
>
>backend httpd server knows it's an authorized frontend proxy connecting
>to 
>it.
>
>Dave

Re: httpd behind proxy

[Дилян Палаузов]

Hello,

Nginx being proxy removes the Etag when sub(stutions) are involved 
(https://forum.nginx.org/read.php?2,242807,242809#msg-242809).

If Nginx is used as proxy and it returns ETags on GET, then most probably the 
backend runs already on https and has the right hostname, so that nginx doesn't 
need any rewritings.

Now, if a client sends Forwarded header and httpd, not being behind a reverse 
proxy, interprets it, replacing the schema and hostname in the answer, e.g the 
URL: in /freebusy/user/... request, then the behaviour of httpd by interpreting 
the header will be correct: the client asked for troubles and got troubles.  
The troubles however do will not happen if httpd is behind a proxy and the 
proxy inserts Forwarded, as only the last Forwarded is supposed to be 
interpreted.  Irgnoring in this case Forwarded, as this is anyway now the case, 
is also correct.

So I propose removing the checks in imap/http_proxy.c:http_proto_host() for 
config_mupdate_server and proxyservers.

Greetings
  Дилян

On June 20, 2018 7:10:45 PM GMT+02:00, Ken Murchison  wrote:
>
>On 06/20/2018 12:23 PM, Dilyan Palauzov wrote:
>> Hello,
>>
>> I want to run cyrus-httpd behind proxy, making it to listen to 
>> 127.0.0.3:80.  It then sends on /freebusy/user/me URL: 
>> http://127.0.0.3/freebusy/user/me , which I don't want.  If I tweak 
>> the front-end, nginx, to rewrite 127.0.0.3 -> my hostname, Nginx is 
>> smart enogh and removes the ETags sent by cyrus/httpd, so this 
>> approach does not work.
>>
>> Then I decided to insert "Forwarded: host=my host; proto=https" 
>> header, however imap/http_proxy.c:http_proto_host handles the 
>> Forwarded header only
>>
>>     if (config_mupdate_server && 
>> config_getstring(IMAPOPT_PROXYSERVERS) &&
>>     (fwd = spool_getheader(req_hdrs, "Forwarded"))) {
>>     /* Proxied request - parse last Forwarded header for proto
>and 
>> host */
>>
>> What is the rationale behind interpreting Forwarded only when 
>> mupdate_server and proxyservers are set?
>
>I don't recall if I had any specific reason in mind when I added that 
>check.  The downside of removing the check is that a client can do as 
>you plan to and can cause the server to change URLs in replies.  I'm
>not 
>a security expert, but this seems like something we don't allow a
>client 
>to do.
>
>I know that we (FastMail) run Cyrus behind nginx and this hasn't become
>
>an issue, unless our ops guys have patched Cyrus or found a different 
>way to handle this in Nginx.  Bron may know, once he wakes up.

Re: use valgrind / Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Дилян Палаузов]

Hello,

On Fri, 2018-07-13 at 10:40 +0200, Michael Menge wrote:
> 
> I have been running mupdate with STARTTLS for a few years with 2.3.x  
> and 2.4.x.
> It didn't crash as predictable as 3.0, but we had some instances where ...

The likely reason, why it hasn't crashеа as predictable is, that the
line DH_free(dh_params); was not part of cyrus-imapd-2.4  . It was
added to the caldav-2.4 branch back in 2014 and from there stayed in
the 3.0 and master branches.  Once a memory leak was found in 2017 on
the cyrus-imapd-2.5 branch, the line was added also there.  So for
fixing this on 2.5 it was a matter of having two persons find the
problem: one finds it and fixes it on caldav-2.4, and somebody else
finds it again on 2.5 two years later and arranges a fix there.

I thought everything from caldav-2.4 went into 2.5.  Now I see this was
not the case.

Please, as a rule, when fixing stuff on master (or caldav-2.4 in this
case), consider whether it is relevant also for the stable versions and
fix it there at the same time.  Postgres or Evolution are good
examples, how this process can be done.

Regards
  Дилян

use valgrind / Re: SIGSEGV in cyrus-imapd 3.0.7 mupdate

[Дилян Палаузов]

Hello Michael,

this is likely either a memory mishandling issue (use after free(),
double free(), invalid read()/write()...), which gets evident if cyrus
is run under valgrind --tool=memcheck.  I run it with

valgrind --num-callers=30 --leak-check=full --track-origins=yes --read-
var-info=yes --show-leak-kinds=all --trace-children=yes --track-fds=yes 
/usr/local/cyrus/bin/master -D &> log-file-memcheck

It runs then very slow, but shows both cause of troubles and memory
leaks.

Another reason can be multi-threaded inconsistencies: mutexes locked in
inconsistent order by differnt threads (while this is not a cause for
crash, it leads to deadlock), mutexes locked in one thread and unlocked
in another or alike.  This can be detected by valgrind/helgrind

valgrind --tool=helgrind --num-callers=30 --leak-check=full --track-
origins=yes --read-var-info=yes --trace-children=yes --track-fds=yes
/usr/local/cyrus/bin/master -D &> log-file-helgrind

Of course, it is useful to have debugging symbols.  Compiling with -O3
migh make things faster, while compiling with -O0 will make it run
considerably slower. If ld (=ld.bfd) is very, very recent (e.g. 2.31),
then link with  -Wl,-z,noseparate-code otherwise valgrind cannot read
the debug info (https://bugs.kde.org/show_bug.cgi?id=395682).

Then it shall be possible to see in one or the other log files what is
relevant for causing SIGSERV, in particular the first place where bad
operation happens, which might be some time before the crash.

Regards
  Дилян

On Mon, 2018-07-02 at 17:24 +0200, Michael Menge wrote:
> Hi,
> 
> we are in the process of setting up our new production mailserver with
> cyrus-imapd 3.0.7 on RHEL 7.5 Servers.
> 
> At the moment we encounter many crashes (SIGSEGV) of the mupdate process on
> the mupdate master instance. As soon as we issue a command that updates
> multiple mailboxes in short time we trigger a SIGSEGV.
> 
> > sam user/zrstes* cyrus all
> 
> Setting ACL on user/zrstes1...OK.
> Setting ACL on user/zrstes1/Mail...OK.
> Setting ACL on user/zrstes1/Mail/drafts...OK.
> Setting ACL on user/zrstes1/Mail/s-spam...OK.
> Setting ACL on user/zrstes1/Mail/sent...OK.
> Setting ACL on user/zrstes1/Mail/trash...OK.
> Setting ACL on user/zrstes1/Mail/v-spam...OK.
> Setting ACL on user/zrstes2...cyrus: lrswipkxtea: no connection to server
> 
> I suspect that at this time the connection from the backend to the  
> mupdate master
> is lost as the mupdate process received the SIGSEGV
> 
> (gdb) bt
> #0  0x7fb49c98 in ?? ()
> #1  0x7fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
> #2  0x7fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
> #3  0x00404075 in conn_free (C=0x7fb4840009f0) at imap/mupdate.c:379
> #4  0x004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
> #5  0x7fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at  
> pthread_create.c:308
> #6  0x7fb4a718fb3d in clone () at  
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> 
> 
> Further information, as DH_free is in the bt, we have NOT configured  
> dh_params in
> our ssl certificate/key or in the imapd.conf
> 
> Kind regards
> 
> Michael Menge
> 
> 
> M.MengeTel.: (49) 7071/29-70316
> Universität Tübingen   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung  mail:  
> michael.me...@zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
> 

Re: shared xDAV resources

[Дилян Палаузов]

Hello,

what is the status of this?  Are the theoretical concerns clarified by
now?

About CALDAV:calendar-home-set:

While after logging in a user is supposed to find the list of the owned
calendars available with CALDAV:calendar-home-set, the specification
says this is an URL and the calendars owned by the user are  below that
URL.  Owned or not-owned does not matter here.

RFC4791 says for CALDAV:calendar-home-set "Identifies the URL of any
WebDAV collections that contain calendar collections owned by the
associated principal resource.". As it is not mandated, that a user has
to own calendars only within a single base URL, many base-URLs can be
returned.

Section 3.1 says: ... That might mean that calendaring data is spread
throughout the repository and mixed with non-calendar data in nearby
collections (e.g., calendar data may be found in /home/lisa/calendars/
as well as in /home/bernard/calendars/, and non-calendar data in
/home/lisa/contacts/).  Or, it might mean that calendar data can be
found only in certain sections of the repository (e.g., /calendar/).  

Any idea what happens, when /dav/calendars is returned in calendar-
home-set?

As having calendar (of bernard and lisa) data under /calendar/ is no
different than having calendar data under /home/bernard/calendars and
/home/list/calendars, the home-set can return either only /calendar/ or
both other URLs.

About caldav-clients not accepting two URLs here... some CalDAV clients
also don't accept the fact that CalDAV resource can be subscribed
without username/password, but they accept subscribing to iCalendar
files without username/password.  So it is a matter of time that caldav
clients adopt.

What is the concept with the data per user from .dav, when the
resources are in a shared namespace?

Regards
  Дилян



On Sat, 2018-04-07 at 18:30 -0300, Anatoli wrote:
> Hi All,
> 
> I'm trying to understand the code responsible for enumerating user calendars 
> (and xDAV resources in general) to try to make the discovery work for shared 
> resources too (currently there's no way to access shared resources with Apple 
> xDAV client implementation, yes with Thunderbird as it doesn't use the 
> discovery mechanism, but instead should be pointed to the exact URL for each 
> calendar). If I understand it correctly, the functionality is in 
> imap/http_caldav.c.
> 
> Could you please point me to the place where the enumeration occurs and 
> briefly mention how the general workflow looks like?
> 
> The client asks for:
> 
> PROPFIND /dav/calendars/user//
> 
> 
> ...
> 
> The server responds with:
> 
> HTTP/1.1 207 Multi-Status
> 
> 
>   
> /dav/calendars/user//
> 
> ...
>   
>   
> /dav/calendars/user//Default/
> 
>   
> ...
> 
> The idea is to include in the returned lists the shared calendars too with 
> the discovery logic based on the IMAP shared folders.
> 
> Below goes the initial exchange between the calendar app on iOS 10.2.6 and 
> Cyrus 3.0.5 when the exact URL (/dav/calendars/shared/) for the shared 
> calendar is provided in the advanced settings of the app (the URL finally 
> resets to the user principals folder (/dav/principals/user/t...@domain.com/) 
> as iOS is pointed to it by Cyrus). In the attached file goes the telemetry 
> for the rest of the communication.
> 
> Thanks,
> Anatoli
> 
> -- t...@domain.com Sun Mar 25 06:05:36 2018
> 
> <1521968736 Accept: */*
> Content-type: text/xml
> Connection: keep-alive
> Content-length: 181
> Host: mail.domain.com
> User-agent: iOS/11.2.6 (15D100) accountsd/1.0
> Prefer: return=minimal
> Depth: 0
> Brief: t
> Accept-language: en-us
> Authorization: Basic ...
> Accept-encoding: br, gzip, deflate
> 
> <1521968736<
> 
>   
> 
> 
> 
>   
> 
> 
> 
> >1521968736>HTTP/1.1 207 Multi-Status
> Date: Sun, 25 Mar 2018 09:05:36 GMT
> Strict-Transport-Security: max-age=600
> Vary: Accept-Encoding, Brief, Prefer
> Preference-Applied: return=minimal
> Content-Type: application/xml; charset=utf-8
> Content-Length: 546
> 
> 
> 
>   
> /dav/calendars/shared/
> 
>   
> 
>   /dav/principals/user/t...@domain.com/
> 
> 
>   
>   
> 
>   
>   HTTP/1.1 200 OK
> 
>   
> 
> 
> <1521968736 Host: mail.domain.com
> Connection: keep-alive
> Accept: */*
> User-Agent: iOS/11.2.6 (15D100) accountsd/1.0
> Accept-Language: en-us
> Content-Length: 0
> Accept-Encoding: br, gzip, deflate
> 
> >1521968736>HTTP/1.1 200 OK
> Date: Sun, 25 Mar 2018 09:05:36 GMT
> Strict-Transport-Security: max-age=600
> Cache-Control: no-cache
> Link: ; rel="server-info"; 
> token="80769c2c66d340ecd178710db26d56b9c4699e3e"
> DAV: 1, 2, 3, access-control, extended-mkcol, resource-sharing
> DAV: calendar-access, calendar-auto-schedule
> DAV: calendar-query-extended, calendar-availability, 
> calendar-managed-attachments
> DAV: calendarserver-sharing, inbox-availability
> DAV: addressbook
> Allow: OPTIONS, GET, HEAD
> Allow: PROPFIND, REPORT, 

Extended LIST RETURN (CHILDREN SUBSCRIBED)

[Дилян Палаузов]

Hello,

right now EXTENDED LIST RETURN (CHILDREN SUBSCRIBED) returns
\Subscribed, as can be seen below, but back in 2015 this was not the
case -- https://bugzilla.gnome.org/show_bug.cgi?id=748636 .

Can you tell me when this was fixed?

Since extended LIST was not working correctly in the past, Gnome
Evolution has not used it when communicating with cyrus, and the normal
LIST by default does not return SPECIAL-USE, so Gnome Evolution didn't
know about applied SPECIAL-USE flags.

Regards
  Дилян

S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
SASL-IR] mail.aegee.org Cyrus IMAP 3.0.7-27-g0507ce576-dirty server
ready
Please enter your password: 
C: A01 AUTHENTICATE PLAIN zzz
S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten
QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT
CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEAR
CH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT
THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1
METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADA
TA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE
DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED
XCONVERSATIONS COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSA
GE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] Success (no
protection) SESSIONID=
Authenticated.
Security strength factor: 0
A LIST "" "Shared Folders/*"
* LIST (\HasNoChildren) "/" "Shared Folders/shared/A"
* LIST (\HasNoChildren) "/" "Shared Folders/shared/B"
A OK Completed (0.002 secs 4 calls)
B LIST "" "Shared Folders/*" RETURN (CHILDREN SUBSCRIBED)
* LIST (\Subscribed \HasNoChildren) "/" "Shared Folders/shared/A"
* LIST (\HasNoChildren) "/" "Shared Folders/shared/B"
B OK Completed (0.002 secs 4 calls)
C LOGOUT
* BYE LOGOUT received
C OK Completed
Connection closed.

Re: Fwd: Re: [cyrusimap/cyrus-sasl] PAM auth causes segfault for unknown reason (#547)

[Дилян Палаузов]

Hello,

the message below does not show the full backtrace at the very last
moment.  One way to create it is to produce a core dump and then open
with gdb the saslauthd executable, the coredump and call "thread apply
all bt f".

More usefull could be to run the process under valgrind/memchech, after
compiling it debug information.

Otherwise it is evident, that the process terminates unexpectedly, but
it is unclear why and where.

Regards
  Дилян

On Mon, 2018-12-10 at 11:46 -0800, jan parcel wrote:
> We have seen severe mutex issues with sasl_client init on super-fast 
> machines.We temporarily solved it by patching openldap to ensure it never 
> calls sasl_client_init more than once,  but when I get out from under my 
> current non-cyrus-sasl load, I had planned to test and submit changes to 
> sasl_*_init to use a static mutext that is pre-initialized, which is what I 
> found online as the solution for LIBRARY issues with passed in or even local 
> mutexts.
> 
> The issue is that the mutex needs a mutex, which needs a mutex, and it's 
> turtles all the way down unless a static mutex is used at that top level.
> 
> For some reason I have not understood, I've been told to not even THINK of 
> doing anything like that to the mutexes used in gssapi.
> 
> 
> On 12/10/2018 7:53 AM, Alexander Sagen wrote:
> > Trying to set up a temporary shared email server to replace an old server 
> > with a dying disk while we move all our customers to a new email solution. 
> > Configured saslauthd the same way it was configured on the old server.
> > Somehow, during the authentication process, saslauthd manages to segfault, 
> > seemingly due to a mutex lock issue.
> > Installed latest sasl2-bin (version 2.1.27) from APT (source: 
> > http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 
> > Packages).
> > Running linux kernel 4.15.0-1021-aws.
> > gdb output during crash:
> > root@mail1:~# gdb --args /usr/sbin/saslauthd -a pam -c -m 
> > /var/spool/postfix/var/run/saslauthd -r -n 0 -d
> > GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
> > Copyright (C) 2018 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later 
> > 
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > ;.
> > Find the GDB manual and other documentation resources online at:
> > ;.
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from /usr/sbin/saslauthd...(no debugging symbols 
> > found)...done.
> > (gdb) run
> > Starting program: /usr/sbin/saslauthd -a pam -c -m 
> > /var/spool/postfix/var/run/saslauthd -r -n 0 -d
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > saslauthd[4767] :num_procs  : 0
> > saslauthd[4767] :mech_option: NULL
> > saslauthd[4767] :run_path   : /var/spool/postfix/var/run/saslauthd
> > saslauthd[4767] :auth_mech  : pam
> > saslauthd[4767] :mmaped shared memory segment on file: 
> > /var/spool/postfix/var/run/saslauthd/cache.mmap
> > saslauthd[4767] :bucket size: 96 bytes
> > saslauthd[4767] :stats size : 36 bytes
> > saslauthd[4767] :timeout: 28800 seconds
> > saslauthd[4767] :cache table: 985828 total bytes
> > saslauthd[4767] :cache table: 1711 slots
> > saslauthd[4767] :cache table: 10266 buckets
> > saslauthd[4767] :flock file opened at 
> > /var/spool/postfix/var/run/saslauthd/cache.flock
> > saslauthd[4767] :master pid is: 0
> > saslauthd[4767] :listening on socket: 
> > /var/spool/postfix/var/run/saslauthd/mux
> > saslauthd[4767] :attempting a read lock on slot: 1501
> > saslauthd[4767] :[login=someu...@example.com] [service=smtp] 
> > [realm=example.com]: not found, update pending
> > saslauthd[4767] :attempting to release lock on slot: 1501
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > __GI___pthread_mutex_lock (mutex=0x20) at ../nptl/pthread_mutex_lock.c:65
> > 65  ../nptl/pthread_mutex_lock.c: No such file or directory.
> > (gdb)
> > saslauthd configuration:
> > START=yes
> > DESC="SASL Authentication Daemon"
> > NAME="saslauthd"
> > MECHANISMS="pam"
> > MECH_OPTIONS=""
> > THREADS=0
> > OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd -r"
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub, or mute the thread.
>  

Re: MMAP performance and using mmap writes

[Дилян Палаузов]

Hello,

> > 3: if your DB is larger than RAM, writing thru mmap is slower than using 
> > write() syscalls. Whenever you
> > access a page for the first time, the OS will page it in. This is a wasted 
> > I/O if all you're doing is
> > overwriting the page with new data.
> 
> I doubt it... especially now we're running on servers with 256Gb of data.  
> These databases are usually under a gigabyte in size.  I also don't think we 
> ever overwrite a page without reading from it first - we're usually updating 
> pointers which we've just had to read.
> 

Are there recommendations on RAM for running cyrus imap, that reflect
this aspect?

Greetings
  Dilyan

Re: db5 / PANIC errors under 2.5.12?

[Дилян Палаузов]

Hello Karl,

man imapd.conf shows the default database types, e.g. check for
annotation_db, duplicate_db…  You likely have there explicit or
implicit berkeley database.  With cvt_cyrusdb you can switch between
database formats.  You use the latter to convert to e.g. twoskip and
then update imapd.conf to state, that the database is in twoskip
format.

On my system the files in imap/db/ were not changed since 2016, I guess
when I upgraded at that time.

Regards
  Дилян

On Wed, 2018-11-21 at 10:15 +, Karl Pielorz wrote:
> This just happened again, on a different server...
> 
> Having spent more time looking at this - the error:
> 
>   Nov 21 07:45:30 imaps[72501]: DBERROR db5: BDB1581 File handles still 
> open at environment close
> 
> I'm guessing "db5" is indeed Berkeley DB5 - so even though in our 
> imapd.conf we don't specify 'berkeley' anywhere (and 'files' on all the .db 
> files confirms - they're all twoskip) - the database being created in 
> '/vol/host/imap/db/' is likely still berkeley.
> 
> How can I change this? - In imapd.conf I can see a myriad of 
> 'seenstate_db', 'mboxlist_db' - but I can't see anything that relates to 
> the files in that DB directory.
> 
> To be honest - I can't see anywhere what that database is actually called 
> (which might help me track down how to change it's format - if you can?)
> 
> Does anyone know what the database that lives off of 'db/' is actually 
> called / used for? - And how I can change it's format, that is presuming 
> the issues I'm seeing are caused by it still being berkeley?
> 
> -Karl

Re: Xapian searches in Cyrus 3.0

[Дилян Палаузов]

Hello Egoitz,

my understanding is, that search always succeeds; indexed search is just 
faster.  Cf. 
docsrc/imap/reference/admin/sop/sqatter.rst .

Triggered squatter does not yet mean, that it stores some results on the hard 
disk.

Regards
  Дилян



On Thu, 2019-01-03 at 17:09 +0100, Egoitz Aurrekoetxea wrote:
> Hi Dilyan,
> 
> I suppose that should be easy. Run Squatter in rolling mode and send you a 
> message in a testing mailbox where no other emails exist. Then, try to find 
> that word :).. shouldn't it?. Or else, send a mailbox with a big word to 
> search and see if it appears in a search... by the way... when you receive 
> that mail, Squatter should be triggered (when in rolling mode) to index that 
> last email...
> 
> Hope it helps!
> Cheers!
> ---
> 
> Egoitz Aurrekoetxea
> Departamento de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
> El 03-01-2019 17:00, Дилян Палаузов escribió:
> > Hello Egoitz,
> > 
> > concerning Xapian see discussion at 
> > https://github.com/cyrusimap/cyrus-imapd/issues/2234 .
> > 
> > I am still struggling how to verify that for new mails the search 
> > information is append, and that for searches this
> > information is utilized.
> > 
> > Regards
> >   Дилян
> > 
> > On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote: 
> > > Good afternoon,
> > > 
> > > I was planning to perform mailboxes squattering in rolling mode. Have you 
> > > had some kind of expience on searching with Xapian and IMAP protocol?. 
> > > Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not 
> > > so advantageous?. How your experiences had been in this sense, with this 
> > > technology?. Is better to use a non Xapian system?. Just, let's say the 
> > > old way of Squatter without Xapian?.
> > > 
> > > Best regards,
> > > -- 
> > > 
> > > Egoitz Aurrekoetxea
> > > Departamento de sistemas
> > > 944 209 470
> > > Parque Tecnológico. Edificio 103
> > > 48170 Zamudio (Bizkaia)
> > > ego...@sarenet.es
> > > www.sarenet.es
> > > 
> > > Antes de imprimir este correo electrónico piense si es necesario hacerlo.

Re: Xapian searches in Cyrus 3.0

[Дилян Палаузов]

Hello Egoitz,

concerning Xapian see discussion at 
https://github.com/cyrusimap/cyrus-imapd/issues/2234 .

I am still struggling how to verify that for new mails the search information 
is append, and that for searches this
information is utilized.

Regards
  Дилян

On Thu, 2019-01-03 at 15:55 +0100, Egoitz Aurrekoetxea wrote:
> Good afternoon,
> 
> I was planning to perform mailboxes squattering in rolling mode. Have you had 
> some kind of expience on searching with Xapian and IMAP protocol?. Perhaps is 
> more designed for JMAP?. Or perhaps, using it with IMAP is not so 
> advantageous?. How your experiences had been in this sense, with this 
> technology?. Is better to use a non Xapian system?. Just, let's say the old 
> way of Squatter without Xapian?.
> 
> Best regards,
> -- 
> 
> Egoitz Aurrekoetxea
> Departamento de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.

To: and qouted strings

[Дилян Палаузов]

Hello,

PUTting a new meeting with

ATTENDEE;CN="X, Y":MAILTO:a@b

sends over 3.0/imap/imip_send_sendmail() to sendmail:
To: X, Y 
which sendmail normalizes to
To: X@fqdm, Y 

The string X, Y must be quoted, as described in RFC5322 Section 3.2.4 “Quoted 
Strings”.

Is there code in cyrus, that does such quoting, when necessary?  I found 
prot_printstring() which works on struct
protstream, but the need is for operating on char* or struct buf.

The codebase will benefit, if somebody does deduplication. There is code on 
several places generating message
boundaries, or calculating the sieve paths for a user.  There is 
lib/prot.c:isQCHAR which seems and sounds very similar
to lib/rfc822tok.c:is_special.  A const array with the value “{ "Sun", "Mon", 
"Tue", "Wed", "Thu", "Fri", "Sat" }” is
defined in imap/http_tzdist.c, imap/httpd.c, and lib/times.c, but a single 
cyrus-wide definition will be sufficient. 
Likewise, a there are several definitions of the weekdays.

Regards
  Дилян

Scheduling VJOURNAL

[Дилян Палаузов]

Hello,

iCalendar forsees for VJOURNAL, VEVENT and VTODO attendees and organizer.

Scheduling Extensions to CalDAV says nothing about VJOURNAL.  iTIP foresees 
VJOURNAL.

In imap/http_caldav.c:caldav_put() has

   switch (kind) {
case ICAL_VEVENT_COMPONENT:
case ICAL_VTODO_COMPONENT:
case ICAL_VPOLL_COMPONENT:
if (organizer)
  DO SOMETHING, possibly sending an email
case ICAL_VJOURNAL_COMPONENT:
DO NOTHING

I do expect, when I upload a VJOURNAL with attendees on different hosts, that 
the participants receive a mail (on each
change).

Do you think it is reasonable to move “case ICAL_VJOURNAL_COMPONENT” before “if 
(organizer)”, so that for local users
HTTP scheduling happens, or shall uploading VJOURNAL with ATTENDEEs always 
trigger emails?

Regards
  Дилян

conversations_expire_days and Xapian search

[Дилян Палаузов]

Hello,

I have an understanding question about searches, conversations, and Xapian in 
3.0.x:

To my knowledge data (headers and body) is filled in a Xapian database.  The 
search with the Xapian database works
however only, if these messages are mapped in conversations_db.  There is a 
conversations_expire_days property with
default 90:

  How long the conversations database keeps the message tracking information 
needed for receiving new messages in
existing conversations, in days.

Does this mean, that by default after 90 days messages are deleted from the 
conversations db and cannot be found anymore
using the Xapian database?  How to setup the days, so that all messages in 
Xapian can be mapped to convesations_db and
therefore found?

Regards
  Дилян

Re: Failed CUnit test in 2.5.12 and 3.0.8

[Дилян Палаузов]

Hello Max,

a possible explanation is the lack of libplain.so or libdigestmd5.so in 
$libdir/sasl2 , coming from cyrus-sasl  .

Until recently, the place where cyrus-sasl/configure && make install put those 
files, when compiled, was different from
the place, where the files were searched at run time.

Regards
  Дилян

On Fri, 2019-02-08 at 17:23 +0300, Max Kosmach wrote:
> Hi, cyrus imap devs!
> 
> I'm trying to build 2.5.12 and 3.0.8 cyrus imapd and in both versions I have 
> same problems with backend suite:
> 
> Suite: backend
>   Test: badhost ...passed
>   Test: badservice ...passed
>   Test: sasl_plain ...Server failed to find requested SASL mechanism "PLAIN"
> FAILED
> 1. ./cunit/backend.testc:198  - CU_ASSERT_PTR_NOT_NULL_FATAL(be)
>   Test: sasl_digestmd5 ...Server failed to find requested SASL mechanism 
> "DIGEST-MD5"
> FAILED
> 1. ./cunit/backend.testc:265  - CU_ASSERT_PTR_NOT_NULL_FATAL(be)
>   Test: multiline_caps ...Server failed to find requested SASL mechanism 
> "PLAIN"
> FAILED
> 1. ./cunit/backend.testc:314  - CU_ASSERT_PTR_NOT_NULL_FATAL(be)
>   Test: oneline_caps ...Server failed to find requested SASL mechanism "PLAIN"
> FAILED
> 1. ./cunit/backend.testc:314  - CU_ASSERT_PTR_NOT_NULL_FATAL(be)
>   Test: starttls ...FAILED
> 1. ./cunit/backend.testc:408  - CU_ASSERT_PTR_NOT_NULL_FATAL(be)
> 
> Is this correct behavior and I can ignore this?
> 
> PS. strace show thak this test search nonexistent testserver.conf file -
> [pid 21654] openat(AT_FDCWD, "/etc/sasl2/testserver.conf", O_RDONLY 
> 
> [pid 21654] openat(AT_FDCWD, "/etc/sasl/testserver.conf", O_RDONLY) = -1 
> ENOENT (No such file or directory)
> [pid 21654] openat(AT_FDCWD, 
> "/usr/lib/x86_64-linux-gnu/sasl2/testserver.conf", O_RDONLY) = -1 ENOENT (No 
> such file or directory)
> [pid 21654] openat(AT_FDCWD, "/usr/lib/sasl2/testserver.conf", O_RDONLY) = -1 
> ENOENT (No such file or directory)
> 

Generating MIME boundaries within cyrus

[Дилян Палаузов]

Hello,

sending a MIME message implies generating multipart boundary (random) string.  
Cyrus does this in:

imap/cyrdump.c:g
enerate_boundary()
imap/httpd.c:write_multipart_body()
imap/http_caldav_sched.c:imip_send_sendmail()
imap/jmap_mail.c:_mime
_make_boundary()
imap/lmtp_siele.c:send_rejection()

Isn’t it feasible to avoid on this occassion code duplication?

Regards
 
Дилян

Prepending Xapian Tiers

[Дилян Палаузов]

Hello,

I set up a Cyrus system with one tier.  I think it works.  The .xapianactive 
files contain 'tiername: 0'.

How can I insert a second tier?

Adding a XYZsearchpartition-default to imapd.conf, together with 
defaultsearchtier: XYZ does not utilize the new directory: it stays empty and 
the .xapianactive files do not get updated to mention the new tier.

Besides, if a message is MOVEd over IMAP, is any optimization utilized, to 
avoid reindexing the message, but just change the address of the document?

Regards
  Дилян

Cyrus Roadmap as VTASK set

[Дилян Палаузов]

Hello,

since Cyrus Imap supports publishing Tasks = Roadmaps (VTODOs over CalDAV or 
over HTTP/iCalendar), will it be feasible to integrate VTODOs into the workflow?

How does a Github issue differ from a VTASK, apart from tracking the references 
in git and the automatic closing?

Regarda
  Дилян

https://imapwiki.org/ImapTest/ServerStatus update for Cyrus Imap 3.0

[Дилян Палаузов]

Hello,

anybody with IMAP expertize willing to update 
https://imapwiki.org/ImapTest/ServerStatus for a current Cyrus Imap
status?

It says right now Cyrus 2.3.12p2 is a non-compliant server.

Regards
  Дилян

Ephemeral files and temporary filesystems

[Дилян Палаузов]

Hello,

the file docsrc/imap/concepts/deployment/performance_recommendations.rst says:

In-memory filesystems are faster then disk filesystems, but are limited in 
space and volatile as well. The following
list includes configuration settings that could make the corresponding 
databases, files and directory trees be created
and maintained on a temporary filesystem.

* ``proc_path``: /dev/shm/cyrus-imapd/proc
* ``mboxname_lockpath``: /dev/shm/cyrus-imapd/mboxname_lock
* ``duplicate_db_path``: /dev/shm/cyrus-imapd/duplicate_db
* ``statuscache_db_path``: /dev/shm/cyrus-imapd/statuscache_db


lib/imapoptions says:
{ "prometheus_stats_dir", NULL, STRING }
/* Directory to use for gathering prometheus statistics.  If specified,
   must be an absolute path.  If not specified, the default path
   $configdirectory/stats/ will be used.  It may be advantageous to locate this
   directory on ephemeral storage. */


doc/examples/imapd_conf/murder-backend.conf, 
doc/examples/imapd_conf/murder-frontend.conf,
doc/examples/imapd_conf/murder-mupdate.conf, 
doc/examples/imapd_conf/normal-master.conf, doc/examples/imapd_conf/normal-
replica1.conf and doc/examples/imapd_conf/normal.conf say:

# Directories for proc and lock files
proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock

# Locations for DB files
# The following DB are recreated upon initialization, so should live in
# ephemeral storage for best performance.
duplicate_db_path: /run/cyrus/deliver.db
ptscache_db_path:  /run/cyrus/ptscache.db
statuscache_db_path: /run/cyrus/statuscache.db
tls_sessions_db_path: /run/cyrus/tls_sessions.db


docsrc/imap/download/packagers.rst says:
Ephemeral files and temporary filesystems
#

In addition to Unix domain sockets and lock files, several databases
used by Cyrus programs may be located in temporary filesystems, such as
those backed by RAM (i.e. tmpfs, md, etc.).  Here's a list of such
files.  In this example, the filesystem ``/run`` is on tmpfs::

proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock
duplicate_db_path: /run/cyrus/deliver.db
statuscache_db_path: /run/cyrus/statuscache.db
ptscache_db_path: /run/cyrus/ptscache.db
tls_sessions_db_path: /run/cyrus/tls_sessions.db
lmtpsocket: /run/cyrus/socket/lmtp
idlesocket: /run/cyrus/socket/idle
notifysocket: /run/cyrus/socket/notify


docsrc/imap/download/upgrade.rst says:
You don't need to copy the following databases as Cyrus 3.0 will
recreate these for you automatically:

* duplicate delivery (deliver.db),
* TLS cache (tls_sessions.db),
* PTS cache (ptscache.db),
* STATUS cache (statuscache.db).

.. note::
You may wish to consider relocating these four databases to ephemeral
storage, such as ``/run/cyrus`` (Debian/Ubuntu) or ``/var/run/cyrus``
or whatever suitable tmpfs is provided on your distro.  It will place
less IO load on your disks and run faster.


docsrc/imap/reference/admin/locations/configuration-state.rst says:

Ephemeral Databases
^^^

Several of the state databases may be located in RAM-backed locations,
such as ``/run`` or ``/var/run`` or where ever your OS roots such
filesystems (i.e. ``tmpfs`` on Linux or ``mfs`` on FreeBSD). Cyrus 3.0 will
recreate the following databases for you automatically:

* duplicate delivery (deliver.db),
* TLS cache (tls_sessions.db),
* PTS cache (ptscache.db),
* STATUS cache (statuscache.db).

Relocating these DBs to ephemeral storage will place less IO load on
your disks and tend to run faster.


Where is the truth?  What is the complete list of things, that will benefit on 
being on a tmpfs?  On one place or
another, STATUS cache, TLS cache, PTS cache, duplicate delivery, proc_path, 
mboxname_path, prometheus_stats_dir are
mentioned as candidates.

Do sockets (lmtpsocket) benefit from being on tmpfs, as only 
download/packagers.rst suggests ?

Putting on a single place all candidates for tmpfs will be good, and using 
consistently /dev/shm or /run or /var/run to
mean tmpfs in all documentation will also be good.

Greetings
  Дилян

Re: Notes 29 July

[Дилян Палаузов]

Hello Ellie,

your personal problems are mentioned for quite some time now.  I do not know 
the details, but I personally wish you to
recover soon permanently.  In any case, this thread is not about you.

The accent is about finding a way where contributions are considered in a 
timely manner.  Currently it is absolutely
unclear and unpredictable, whether submitted fixes, being clarifying the 
documentation, adding features or fixing bugs
will be consideted for intergration, so that other users can benefit from them. 
 Whatever conditions are fulfilled, it
is still unpredictable.  This uncertainty questions whether it makes sense to 
contribute upstream to make cyrus imap
better.

As an example, uploading a meeting with
ATTENDEE;CN="A, B":mailto:b.a@examle.orggenerates an email/iMIP invitation with:
To: A, B 
which is converted by the MTA/MSA in
To: A@thisdomain, B 
which is wrong.  Correct is:
To: "A, B" 
https://github.com/cyrusimap/cyrus-imapd/pull/2693 contains a fix from April 
2019.

The question is not what happened with this particular example, but how to 
avoid such stalls in the future.

Besides if a long time passes between a notice on github and a reaction, the 
one who wrote the notice might not remember
anymore the details and this complicates the situation more.

A further challenge, that needs to be approached, on which contributors have no 
impact, is that most users do not run
the “master” version, but a stable one and in turn the provided patches are 
towards the stable version.   In some parts,
the master and stable versions have diverged significantly and continue to 
diverge further.

Regards
  Дилян



On Tue, 2019-07-30 at 09:28 +1000, ellie timoney wrote:
> On Mon, Jul 29, 2019, at 11:10 PM, Anatoli via Cyrus-devel wrote:
> > There are a lot of issues with both 3.0, 3.1 and 3.2 tags (some even have 
> > 2.5 tag), it's not clear which are scheduled to be closed for which release.
> > 
> 
> The tags for existing versions (i.e. 2.4, 2.5, 3.0, 3.1) indicate which 
> versions are known (or suspected) to be affected by those issues, but they 
> don't indicate any particular release schedule for fixes.
> 
> The 3.2 tag currently indicates stuff we definitely need to fix/implement on 
> master before forking the 3.2 stable branch, but after this occurs the 3.2 
> tag will continue like the rest.
> 
> The main person working through the github issues tracker is myself, but I've 
> been working reduced hours for medical reasons for quite a while so there's 
> some weeds growing.  Sorry about that!

#2373 (Shared xDAV (CalDAV/CardDAV) resources are not discoverable) / Re: time for cyrus-imap v3.2?

[Дилян Палаузов]

Hello,

> *  #2373 (Shared xDAV (CalDAV/CardDAV) resources are not discoverable).
> Dilyan Palauzov sent a diff for this in the github repo and there was a
> discussion with Ken on possible implementations (shared xDAV resources):
> https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-May/004263.html.
> I guess it had enough progress to try to close it.

as far as I remember there is some DAV sharing progress in the 3.2 upcoming 
release, but I do not know the details.  I
guess it handles just scheduling of events of shared/unowned calendars, but not 
the discovery.  At CalDAV level the
options for discovery of foreign collections are:
- clients (CUAs) support for sharing resources the model described at 
https://evertpot.com/webdav-caldav-carddav-sharing/ , or 
- CUAs they discover other users by utilizing the  
DAV:princapal-property-search to discover the home-sets of another
user,  from there obtain the accessible calendars, and the CUA memorizes to 
which calendar it has subscribed.

I am not aware of any CUA which does anything of the above. The third option is

- CUAs do not change, but the server changes.  Once the properties of a 
home-set (user4/#calendars or
user7/#addressbooks ) have READ/LOOKUP righth, these home-sets are returned to 
the CUA, when CUA asks for the accessible
home-sets.  On the next step the CUA iterates over all returned home-sets and 
fetches the collections to which it has
access (READ/Lookup ACL).  If there is a calendar that does not belong to any 
user (created with hack command over
IMAP), then this calendar could be a part of a fictional 
/dav/calendars/user/anonymous@domain/ home-set.  Such no-owner
calendars have to my knowledge no-advantage over calendars belonging to the 
user/ namespace.

The advantage of the last method is, that clients do not have to add any new 
features.  The disadvantage is, that all
users see all calendars instantly they have access to, and these can be too 
much.  In particular, clients that have
implemented the first two models/workflows, do not have any advantage over 
clients which have not implemented that two
workflows.

Greetings
  Дилян

8074597e mailbox.c: release cache files when locking index

[Дилян Палаузов]

Hello,

I have the problem that on 3.0 sometimes the cyrus.cache gets truncated to 4 
bytes (usually but not always by
cyr_expire) and then it takes on Inbox very, very much IO and time to 
reconstruct cyrus.cache, and after a while
cyrus.cache gets trucated again.

Does this commit 
https://github.com/cyrusimap/cyrus-imapd/commit/e8074597e84cfb62cc fix the 
problem, is it useful for
3.0, does somebody see in syslog similar problems?  What problem does this 
commit solve?

Greetings
  Дилян 

Re: The master janitor goes crazy / Re: Debugging Deadlocks

[Дилян Палаузов]

Hello Ellie,

this is exactly what I see (countless pselect calls), but I have as second 
parameter of pselect a much larger array.  I
just observed that on killing master, it terminates all cyrus processes but two 
(httpd and notifyd).  Then I try to
connect to that processes (gdb).  This does not work, however, since the 
processes are moved to zombie status.

Greetings
  Дилян

On Thu, 2019-11-28 at 10:34 +1100, ellie timoney wrote:
> Saw something similar just now when I killed a cassandane run off 
> prematurely. One cyrus master process wound up spinning like this:
> 
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> 
> 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> 1221  janitor_position = janitor_position % child_table_size;
> (gdb) bt
> #0  0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> #1  0x555ac712a67a in main (argc=10, argv=0x7ffdc1fe78b8)
> at master/master.c:2812
> 
> Haven't dug further yet, but it looks similar to your report
> 
> On Wed, Nov 27, 2019, at 9:17 AM, ellie timoney wrote:
> > Can you strace the master process next time it's spinning at 100%?  
> > What is it doing at that time?
> > 
> > On Tue, Nov 26, 2019, at 1:29 AM, Дилян Палаузов wrote:
> > > Hello,
> > > 
> > > > I run cyrus imap 3.0.x with some private changes.
> > > > 
> > > > Sometimes when stop the master process, the master process utilizes one 
> > > > CPU core to 100% for 5 minutes.  After the fifth
> > > > minute, systemd enforces kill -9. When I attach to the maste process, I 
> > > > see that it some janitor does some work, but I
> > > > have not checked the details.  Has anybody experienced this?
> > > 
> > > I run cyrus imap.   At some moment I recompile and reinstall the 
> > > binaries, which in theory means that the binaries
> > > detect this change and restart theirselves.  At some moment I call 
> > > "systemctl stop cyrus-imap" which I guess sends
> > > SIGTERM to the master process.   Then the CPU utilization of the master 
> > > process goes to 100%.  In the systemd service
> > > file I have TimeoutStopSec=320 . After this time, the master process 
> > > continues running and systemd sends 9/SIGKILL.  It
> > > is not necessary that on re-installing the binaries, and then shutting 
> > > down the CPU goes to 100%: it is possible that
> > > the CPU goes to 100%, without reinstalling (and thus triggering 
> > > self-restarting) of the imapd/httpd binaries.
> > > 
> > > It is often, but not always, that this 100% CPU loop is entered on 
> > > shutdown.
> > > 
> > > I have a webmail client and to speedup things it uses SquirrelMail's 
> > > IMAP Proxy (http://www.imapproxy.org/ a Caching
> > > IMAP proxy).  It is recommended in the installation manual of 
> > > Horde/IMP.  The IMAP caching proxy connects to
> > > 127.0.0.2:143 (and is therefore permitted to skip the TLS overload).  
> > > In master conf I have a line
> > > “imaplocal cmd="imapd -C /usr/local/etc/cyrus/imapdlocal.conf" 
> > > listen="127.0.0.2:imap" prefork=0”.
> > > 
> > > When the CPU goes to 100% on shutdown I connect with gdb to the master 
> > > process.  Below is the full backtrace.  Does
> > > somebody have an explanation why the master process enters a never 
> > > ending loop?
> > > 
> > > I do not say that all above information has to be involved in the 
> > > anwer.  Has somebody else experienced this effects? 
> > > Any suggestions how to investigate this deeper?
> > > 
> > > Greetings
> > >   Дилян
> > > 
> > > ---
> > > warning: Could not load vsyscall page because no executable was 
> > > specified
> > > Reading symbols from /usr/local/libexec/master...
> > > Attaching to program: /usr/local/libexec/master, process 9247
> > > Reading symbols from /usr/local/lib/libcyrus_min.so

cyrus.cache causes IOERROR: offset greater than cache size

[Дилян Палаузов]

Hello,

sometimes I get in the logs these messages:

Dec 01 01:30:50 mail cyrus/cyr_expire[13952]: IOERROR: offset greater than 
cache size 5243456 2288(0)
Dec 01 01:30:50 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 40568 (System I/O error)
Dec 01 01:30:54 mail cyrus/cyr_expire[13952]: IOERROR: offset greater than 
cache size 5244620 2288(0)
Dec 01 01:30:54 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 40569 (System I/O error)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: offset greater than 
cache size 5247552 2288(0)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 40571 (System I/O error)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 15463 (Mailbox format
corruption detected)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: cache entry truncated 
1072 1835101728 2288(0)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 15464 (System I/O error)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: cache entry truncated 
2080 1131376244 2288(0)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 15465 (System I/O error)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: offset greater than 
cache size (priority)3136 2288(0)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: invalid cache record for 
user.u1 uid 15466 (System I/O error)
Dec 01 01:30:55 mail cyrus/cyr_expire[13952]: IOERROR: offset greater than 
cache size (priority)3976 2288(0)

Often it is connected to cyr_exipre, but not always. It can be also lmtpd.

When a cyrus.cache inconsistency is detected, the cyrus.cache is rebuild. This 
means reading a lot of files from the
disk.  During the reconstruction some locks are active, so effectively a lot of 
processes (lmtpd, imapd, httpd) are
started and all of them wait for the lock to be released.  This cache rebuild 
happens sometimes (perceived) very often. 
The problem is that on slow hard disks this repack operation can take hours and 
cyr_expire runs for hours.

My reading of the code is that new records are only appended to cyrus.cache and 
there is some lock ensuring the
consistency of the append operation.

I have not invested that much time in reading the code.  How is expunging 
supposed to happen in regards of cyrus.cache? 
Is the on unlink()ing any message the cyrus.cache always supposed to be 
repacked or where is the code for removing
entries from cyrus.cache?  How can I debug the cause of the invalid cache 
record?  

I assume that the cached records are kept, until the corresponding message file 
is removed from the disk.

The cyr_expire output also contains:

Dec 01 01:37:37 mail cyrus/cyr_expire[13952]: IOERROR: conversations_audit on 
load: /var/imap//user/s/s2.conversations
B25572d90ed3363c1
 0 (713535 1 0 0 () ((18 713534 1 1 0)) () 
Pleaseconfirmyourregistrationnow. 0 ())

What am I supposed to do with this message?

Regards
  Дилян

The master janitor goes crazy / Re: Debugging Deadlocks

[Дилян Палаузов]

Hello,

> I run cyrus imap 3.0.x with some private changes.
> 
> Sometimes when stop the master process, the master process utilizes one CPU 
> core to 100% for 5 minutes.  After the fifth
> minute, systemd enforces kill -9. When I attach to the maste process, I see 
> that it some janitor does some work, but I
> have not checked the details.  Has anybody experienced this?

I run cyrus imap.   At some moment I recompile and reinstall the binaries, 
which in theory means that the binaries
detect this change and restart theirselves.  At some moment I call "systemctl 
stop cyrus-imap" which I guess sends
SIGTERM to the master process.   Then the CPU utilization of the master process 
goes to 100%.  In the systemd service
file I have TimeoutStopSec=320 . After this time, the master process continues 
running and systemd sends 9/SIGKILL.  It
is not necessary that on re-installing the binaries, and then shutting down the 
CPU goes to 100%: it is possible that
the CPU goes to 100%, without reinstalling (and thus triggering 
self-restarting) of the imapd/httpd binaries.

It is often, but not always, that this 100% CPU loop is entered on shutdown.

I have a webmail client and to speedup things it uses SquirrelMail's IMAP Proxy 
(http://www.imapproxy.org/ a Caching
IMAP proxy).  It is recommended in the installation manual of Horde/IMP.  The 
IMAP caching proxy connects to
127.0.0.2:143 (and is therefore permitted to skip the TLS overload).  In master 
conf I have a line
“imaplocal cmd="imapd -C /usr/local/etc/cyrus/imapdlocal.conf" 
listen="127.0.0.2:imap" prefork=0”.

When the CPU goes to 100% on shutdown I connect with gdb to the master process. 
 Below is the full backtrace.  Does
somebody have an explanation why the master process enters a never ending loop?

I do not say that all above information has to be involved in the anwer.  Has 
somebody else experienced this effects? 
Any suggestions how to investigate this deeper?

Greetings
  Дилян

---
warning: Could not load vsyscall page because no executable was specified
Reading symbols from /usr/local/libexec/master...
Attaching to program: /usr/local/libexec/master, process 9247
Reading symbols from /usr/local/lib/libcyrus_min.so.0...
Reading symbols from /lib/libuuid.so.1...
Reading symbols from /usr/local/lib/libgssapi_krb5.so.2...
Reading symbols from /usr/local/lib/libkrb5.so.3...
Reading symbols from /usr/local/lib/libk5crypto.so.3...
Reading symbols from /usr/local/lib/libcom_err.so.3...
Reading symbols from /usr/local/lib/libkrb5support.so.0...
Reading symbols from /usr/local/lib/libpcreposix.so.0...
(No debugging symbols found in /usr/local/lib/libpcreposix.so.0)
Reading symbols from /usr/local/lib/libpcre.so.1...
(No debugging symbols found in /usr/local/lib/libpcre.so.1)
Reading symbols from /usr/local/lib/libxml2.so.2...
Reading symbols from /usr/local/lib/liblzma.so.5...
(No debugging symbols found in /usr/local/lib/liblzma.so.5)
Reading symbols from /usr/local/lib/libical.so.3...
Reading symbols from /usr/local/lib/libicalss.so.3...
Reading symbols from /usr/local/lib/libicalvcal.so.3...
Reading symbols from /usr/local/lib/libicui18n.so.63...
Reading symbols from /usr/local/lib/libicuuc.so.63...
Reading symbols from /usr/local/lib/libicudata.so.63...
(No debugging symbols found in /usr/local/lib/libicudata.so.63)
Reading symbols from /usr/local/lib/libsqlite3.so.0...
(No debugging symbols found in /usr/local/lib/libsqlite3.so.0)
Reading symbols from /usr/local/lib/libz.so.1...
(No debugging symbols found in /usr/local/lib/libz.so.1)
Reading symbols from /lib64/libm.so.6...
Reading symbols from /lib64/libdl.so.2...
Reading symbols from /lib64/libpthread.so.0...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libc.so.6...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /lib64/libresolv.so.2...
Reading symbols from /usr/local/lib/libdb-18.1.so...
Reading symbols from /usr/local/lib64/libstdc++.so.6...
Reading symbols from /usr/local/lib64/libgcc_s.so.1...
Reading symbols from /usr/local/lib64/libssl.so.1.1...
Reading symbols from /usr/local/lib64/libcrypto.so.1.1...
Reading symbols from /lib64/libnss_db.so.2...
Reading symbols from /lib64/libnss_files.so.2...
Reading symbols from /lib64/libnss_dns.so.2...
0x00405406 in child_janitor (now=...) at master/master.c:1192
1192janitor_position = janitor_position % child_table_size;
™(gdb) bt f
  Id   Target Id Frame 
* 1Thread 0x7f6a08759780 (LWP 9247) "master" 0x00405406 in 
child_janitor (now=...) at master/master.c:1192
#0  0x00405406 in child_janitor (now=...) at master/master.c:1192
i = 9299
p = 0x4132e0 
c = 0x0
#1  0x00409dd7 in main (argc=4, argv=0x7ffea3075108) at 
master/master.c:2600
i = 14
ready_fds = 3
total_children = 11
tv = {
  tv_sec = 

Debugging Deadlocks

[Дилян Палаузов]

Hello,

I run cyrus imap 3.0.x with some private changes.

Sometimes when stop the master process, the master process utilizes one CPU 
core to 100% for 5 minutes.  After the fifth
minute, systemd enforces kill -9. When I attach to the maste process, I see 
that it some janitor does some work, but I
have not checked the details.  Has anybody experienced this?

I have very few users, but one of the users (me) uses many client 
simuitaneously.  Lets say two IMAP clients, making 4-6 
connections in parallel and three CalDAV clients, doing estimated 3-6 
connections in parallel.  The httpd process is
behind a proxy and most of the time the proxy server manages to serialize the 
requests, and in fact a single httpd
process handles the requests.  At least it is not visible that under normal 
circumstances there is a second running
httpd process.  Under normal circumstances I see also a single lmtpd process 
and many imapd processes.

On some days I observe that the IMAP client cannot obtain list of new messages, 
it just times out.  This could because
of deadlocks, but it can be because on that particular day the IO is extremely 
slow and thus the problem is not withn
cyrus.  Sometimes I observe afterwards that tha INBOX index is being rebuild.  
Sometimes, after the INBOX index is
rebuild things start working.

So on such days I suspect that there is some deadlock.  Lets say, if there are 
two or more long-term running lmtpd
processes, then I suspect a deadlock.  What approach can use to find where the 
deadlock is and how can get rid of it?

I can attach to a process with STRACE, get the current backtrace and variable 
values with GDB and I can see (eg. with
LSOF) which files are opened in which mode.  But I do not know what to look 
for.  Or rather, when I try investigating,
almost always I see a process rebuiding my INBOX index and after waiting, 
waiting, waiting, eventually the INDEX is
rebuild.  How can I find out why the index was broken?

Greetings
  Дилян

Re: yearly release cycle

[Дилян Палаузов]

Hello!

This is a very good idea!  In particular it makes the gap between development 
code and stable code smaller.  Thus fixes
for the stable code will be very similar to fixes on the development code.

Of course, providing fixes, like optimizations, makes only sense if it is 
predictable whether the changes will be
integrated in reasonable time.

The email of Quanah Gibson-Mount from 25 July about the general policy on 
integrating patches in Cyrus SASL is not
answered.

Will the time–based release policy also apply to Cyrus SASL?

The documentation of Cyrus IMAP, in its invisable parts, needs some tweaking, 
like the Table Of Content shall be loop-
free.  In March I submitted a fix at 
https://github.com/cyrusimap/cyrus-imapd/pull/2703 which is still pending .  
Today
I have forgotten the detalis, so even if somebody starts integrating this and 
has questions, I am not willing to reread
again how Sphinx works (I have not used Sphinx since then) and digest why I did 
things in a particular way.

Each release announcement encourages contributions to the documentation.

Regards
  Дилян

On Fri, 2019-12-13 at 09:59 -0500, Ricardo Signes wrote:
> Hey, remember last month when I asked about releasing Cyrus v3.2?
> 
> That thread had some more conversation about what needs to get done before 
> v3.2, and I wanted to come back to it and turn some things on their head.
> 
> Right now, we’re talking about Cyrus releases being feature-bound. “We’ll 
> release v3.2 when feature X is done.” I think we’re not being well-served by 
> that. As feature X is delayed (for various reasons that we can’t easily 
> eliminate), it doesn’t just delay the feature, but also all the other minor 
> bugfixes and optimizations that we’ve made in the master branch. Also, it 
> sets up the idea that we delay releases for the sake of fixes, instead of 
> releasing the fixes that are ready.
> 
> That is: every additional criteria for a new release is another doorway to 
> delay. Instead of opening those doors, I would rather try to eliminate all of 
> them.
> 
> I propose that instead of tying releases to milestones, we tie them to the 
> calendar. For the sake of full disclosure: I am modeling this suggestion on 
> the release cycle of perl, which I ran for several years. I found the process 
> more than satisfactory, then.
> 
> A new unstable release of Cyrus is made every month. We promise only that it 
> compiled and passed the Cassandane test suite on the release manager’s 
> computer. It might contain regressions from previous unstable releases, it 
> might have crashers or corruptors. We try to avoid any of these, but the goal 
> here is a snapshot for easy month-to-month testing. These are the 
> odd-middle-digit releases. (3.3.x)
> 
> A new major release of Cyrus is made every year. We will have tested it on as 
> many configurations as we can readily test. We will have, some time before 
> the release, frozen the branch for risky changes, to reduce churn. In the 
> meantime, new work lives in feature branches. (The changelogs from each 
> unstable release provide a good basis for the whole-year changelog!) These 
> are the even-middle-digit third-digit-zero releases. (3.4.0)
> 
> A new maintenance release of Cyrus is made for the last two stable releases 
> when there are enough fixes to critical bugs to warrant it. These are the 
> even-middle-digit third-digit-nonzero releases (3.4.1)
> 
> For the above to work, some more properties need to be maintained.
> 
> Maintenance releases should be no-brainers to install, so they must only fix 
> regressions, crashers, security vulnerabilities, and the like. This means 
> that once you’re on 3.4.0, you can always upgrade within the 3.4 series with 
> a minimum risk. It also means you get no optimizations, features, and the 
> like.
> 
> Major releases must clearly document any incompatible changes or upgrade 
> steps required. Because non-regression bugfixes aren’t backported, we want 
> everyone to be able to upgrade from major release to major release, so 
> incompatible changes must be kept to a minimum.
> 
> In part, this is just “don’t kill off a feature people use just because it’s 
> a little annoying.” The more important one is “don’t introduce half-baked 
> things that might need to change,” because people will come to rely on them 
> before you get the updates finished. For features that will require multiple 
> years to get right, they have to go behind a default-off configuration 
> option. I’d strongly suggest they all have a uniform substring like 
> “unstable”. That way, when a complaint comes in that the behavior of JMAP 
> calendaring has changed, we can reply, “well, to use it, you had to turn on 
> the unstable_jmap_calendaring” option.
> 
> If we go with this policy, we’ll need to…
> 
> identify what issues are blockers to v3.2.0, meaning they’re regressions from 
> v3.0 and would reasonably prevent someone from upgrading; this does not 
> include all known 

IMAP NOTIFY (RFC 5465) ?

[Дилян Палаузов]

Hello,

is RFC 5465 (IMAP NOTIFY (https://tools.ietf.org/html/rfc5465.html) supported 
in 3.0 as stated at 
https://www.cyrusimap.org/imap/rfc-support.html ?

Per RFC 5465 page 7, the server-side support is announced with “ S: * OK 
[CAPABILITY IMAP4REV1 NOTIFY]”.  The CAPABILITY
answer is generated in imap/imapd.c using “static struct capa_struct 
base_capabilities[]” in capa_response().  The
latter contains no code to generate NOTIFY.

Moreover, the event_notifier in imapd.conf suggests utilizing it for EVENT 
NOTIFICATIONS,but the possible options for
notifyd(8) 
https://www.cyrusimap.org/imap/reference/manpages/systemcommands/notifyd.html 
are null, log, mailto, zephyr
and external: none of which is applicable to IMAP NOTIFY as substitution of 
IMAP IDLE.

That leaves me with the impression that the documentation is misleading about 
IMAP NOTIFY support.

Greetings
  Дилян

When is imapproxy usefull?

[Дилян Палаузов]

Hello,

the documentation on performance of Horde/Imp: 
https://www.horde.org/apps/imp/docs/PERFORMANCE recommends installing an imap 
proxy , e.g. the one from https://squirrelmail.org/download.php which ceased 
deveopment decades ago.  The idea is that horde/imp/the-webmail-client connects 
to the imap-proxy and this should make everything faster.

Does such an imap-proxy make things faster for a webmail client?  Is 
nginx-imap-proxy or anything else better than squirrelmail-imap-proxy?

Greetings
  Дилян

cyrus-devel@lists.andrew.cmu.edu does not do DMARC From-mungling

[Дилян Палаузов]

Hello,

recently on this mailing list (cyrus-devel@lists.andrew.cmu.edu) there were 
messages From: Anatoli .  For
that domain DMARC policy=reject applies.  So the MLM is supposed to mungle the 
From: domain, but it does not do so.  In
turns the emails from that recepient over the mailing are subject to reject at 
the final mailboxes.

It is pure accident that I found this out (just seeing replies from Ken to 
emails which I have not received, more emails
in the online archives, which I have not received; generated DMARC failure 
reports, which were sent nowhere, as the
DMARC report has no rua= and ruf= tags).  I can offer mailing list hosting at 
@lists.aegee.org with DMARC mungling for
free.

RFC 5465 IMAP NOTIFY capable client is Gnome Evolution: 
https://gitlab.gnome.org/GNOME/evolution-data-server/issues/183#note_730696

Greetings
  Dilyan

Deleting stale git-branches

[Дилян Палаузов]

Dear all,

when I work on the command line with git and want from git to
autocomplete the words “sieve” or “master”, git does not know if I mean
the branches “sieve-1.2/master” or the directories “sieve/master”.
 This ambiguity makes thing unnecessary complicated.

• Please delete all stale branches, like sieve-1.2 (last updated in
1999).

• Either rename the master branch, or rename the master directory, so
that “git log -p ma" is unambiguous.

Greetings
  Дилян

“ctl_mboxlist -d | ctl_mboxlist -u” makes some fields disappear

[Дилян Палаузов]

Hello,

for turning on improved_mboxlist_sort I turned off cyrus imap,  called
`ctl_mboxlist -d > FILE`, renamed mailboxes.db to mailboxes.db2,
switched in imapd.conf 'improved_mboxlist_sort:1', and called
`ctl_mboxlist -u < FILE`, cf 
https://www.cyrusimap.org/imap/developer/thoughts/improved_mboxlist_sort.html
 .

This created a new mailboxes.db file, which is 2.33 times smaller, that
the original (renamed).

The database format is twoskip.  

Converting both the old and new mailboxes.db to flat with
  $cvt_cyrusdb MAILBOXES.dbX twoskip MM flat
and comparing the results, the old file contains


user.ttt.#addressbooks.MMM  %(A %(ttt lrswipkxtecdan) I a0ab91ff-
6de8-4961-b9cf-646b8037e87a P default T a V 1562330471 F 674 M
1599389319)
 
whereas the new file contains

user.ttt.#addressbooks.MMM  %(A %(ttt lrswipkxtecdan) P default T a
M 1601752550)

The fields:
I a0ab91ff-6de8-4961-b9cf-646b8037e87a 
V 1562330471
F 674

are missing from the new file.

Shall I be worried?

Greetings
  Дилян

Re: expanding allowsetacl

[Дилян Палаузов]

Hello Ricardo,

Horde/IMP allow the user to edit their own IMAP ACLs. Unfortunately I do not 
know of any WebDAV client which users can
utilize to edit their own ACLs.  For WebDAV editing ACLs is one way to share (a 
caldav) collection.

Why is there a plan to kill the ability of users to set their own ACLs?

Greetings
  Дилян

On Wed, 2020-06-24 at 10:57 -0400, Ricardo Signes wrote:
> Behold this commit:
> 
> commit da8305164877735ba29b078151c70455f1aa6eea
> Author: Bron Gondwana 
> Date: Wed Oct 30 14:13:38 2019 +1100
> 
>  imapd: allow disabling the SETACL command
> 
> I'm not sure what the intent here was, but I assume it was related to our 
> plan to kill of the ability of users to set
> their own ACLs.  If so, I think we need two small changes which I'd like to 
> get out pretty soon.
> 
>    1. relax the restriction so the admin user can still use SETACL
>    2. tighten the restriction so it also restricts DELETEACL
> 
> That's it!  Then we'll move on to rolling that out and cleaning up users' 
> existing ACLs.  I have tentatively made a
> task for this.
> 
> Bron:  I just want to make sure we're not stepping on work intended for 
> something else!
> 
> -- 
> Ricardo Signes (rjbs)
> CTO, Fastmail
> 

Two communication channels

[Дилян Палаузов]

Hello,

for Cyrus Imap there are now two communication channels:
cyrus-devel@lists.andrew.cmu.edu and
https://github.com/cyrusimap/cyrus-imapd/issues .  Based on what
criteria shall one use the one or the other channel, to get the things
tackled with higher probability?

Yesterday there were questions on cyrus-devel concerning the role of
pcre and zeroskip.

The dependency on pcre is discussed at
https://github.com/cyrusimap/cyrus-imapd/issues/2891 : as far as I
remember pcre is supposed to be used for UTF-8.

The zeroskip state is discussed at
https://github.com/cyrusimap/cyrus-imapd/issues/3007#issuecomment-597431775
.

I propose that somebody unifies these comminication channels, e.g. by
posting everything from github to cyrus-devel or alike.  This will lead
to some limited extend avoiding discussing the same thing on github and
on cyrus-devel.

Opinions welcome.

Greetings
  Дилян