Hi Anatoli, > Ellie, the patch that you've sent for the backup linking problem > doesn't solve the issue Does anything different happen, or is it exactly the same as without it? Can you provide output? I'm pretty sure my patch is on the right track, but there's almost certainly a detail I've missed since I can't reproduce it myself. The other thing I wanted to ask was if you could run make with "-k". I have a hunch the other backup tools will currently fail to link for you in the exact same way, but if make stops at the first failure then we don't get to see that, nor whether they fail or succeed for some other reason (which might provide a clue). > There is a detailed explanation of the issue here: > https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries > and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition. Thanks for tracking down those links, I'll look through them. > I've already proposed to Ken a slightly different patch than yours > (Makefile.am.patch, attached to this email is its version with > adaptations to apply smoothly to the latest master commit 9c8e8ea) > that works well under these rules. > > The changes are: to add SSL_LIBS to the list of the libs for > libcyrus_backup (currently there are none) and for the second error > (that was mentioned in the initial mail about these problems) to add > lib/libcyrus.la to the list of the libs for cyr_backup. >
Yep, I saw that patch. The reasons my patch is different are that: * libcyrus_backup.la is a no-install library. At present, I expect anything linking against it to also link against anything it needs (hence not having any LDADD of its own). It sounds like this needs to change because of the SHA1_Update call in the library. * libcyrus.la is in $LD_BASIC_ADD, which is in $LD_UTILITY_ADD, and thus is already on the list for cyr_backup and the other backup tools (but clearly in the wrong order) * I want, as much as possible, to avoid having to manually specify lists of library dependencies for build targets. (That gets unmaintainable very quickly) So I'm looking for a way to get the libraries specified in the correct order, portably, and with minimal hand maintenance of dependency lists. Given that the existing system works for everything but the backup tools (with their internal API library), and given that literally nobody but me uses the backup tools at the moment (afaik), there's an opportunity here to integrate them correctly, if a way can be found to do so. If we just manually add them to the lists, then next time something like this shows up we'll still have no idea how to fix it properly > I made a patch (configure.ac.patch, attached) > [...] > Please apply the fixes as you see fit. Thanks, those look good to me. I'll push it up, but I'll split it into separate commits for each piece (sieve/RSCALE/com_err) for historical clarity. I'll also fix the "distribuion" typo while I'm in there, that's been annoying me for ages. :P Cheers, ellie On Tue, May 17, 2016, at 10:21 AM, Anatoli via Cyrus-devel wrote: > Hi All, > > Ellie, the patch that you've sent for the backup linking problem > doesn't solve the issue, but you're right, I've just dug deeper: it's > a matter of order of the library inclusion options and it's because in > Ubuntu starting from 11.04 (and some versions of Debian too) the gcc > invokes ld with --as-needed flag that makes the order of libraries > quite important. There is a detailed explanation of the issue here: > https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries > and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition. > > Also it looks like newer versions of gcc/ld (mine is 4.8.4/2.24) treat > the order of includes much more strictly than previous versions on > most platforms. > > In a nutshell, a binary/library (A) which calls an external function > defined in another library (B) should appear *before* the library (B) > containing the function. The reason is that gcc/ld with --as-needed > flag optimizes symbols that aren't referenced, so if it sees library B > first, and no one has referenced it at that point then it won't link > in anything from it at all. This is what happens with -lcrypto going > before libcyrus_backup.a: the SHA1_Update function is only used inside > libcyrus_backup.a but the linker has already stripped it from - > lcrypto. If we include -lcrypto again *after* libcyrus_backup.a, > everything builds fine. > > > I've already proposed to Ken a slightly different patch than yours > (Makefile.am.patch, attached to this email is its version with > adaptations to apply smoothly to the latest master commit 9c8e8ea) > that works well under these rules. > > The changes are: to add SSL_LIBS to the list of the libs for > libcyrus_backup (currently there are none) and for the second error > (that was mentioned in the initial mail about these problems) to add > lib/libcyrus.la to the list of the libs for cyr_backup. > > So, instead of expecting the include libs, that are required for > certain parts of the binary/lib we're building, to make random > appearance in the gcc line (in sometimes incorrect order under the strict/as- > needed rules), we add in the Makefile.am the needed libs for each and > every lib/block of code we use to compose the final output. As a > result, at the linking stage they appear in the correct order. > > > > > Warning (mostly harmless): No library found for -lcrypto > > Indeed, it's a Makefile.PL warning as my perl installation LD path is > not complete (doesn't include /usr/lib64 where crypto/ssl libs are). > It should be fixed locally in my perl deployment. In any case it's > indeed harmless as gcc/ld knows about this path and this is where > it's used. > > > > > The yacc/lex dependency is actually not checked. Better to say > configure tries to determine what yacc/lex progs are installed, but if > it founds none, it continues as if everything's OK. The config.log > I've sent you was for the build with the packages installed (otherwise > you can't proceed with the make stage). Without the packages, in the > config.log I get: > > configure:19090: checking for bison > configure:19120: result: no > configure:19090: checking for byacc > configure:19120: result: no > configure:19133: checking for flex > configure:19163: result: no > configure:19133: checking for lex > configure:19163: result: no > > but nothing more. So, configure reports no warnings/errors with > respect to the missing tools and then with make I get: > > ./ylwrap: line 175: yacc: command not found > make: *** [sieve/addr.c] Error 127 > make: *** Waiting for unfinished jobs.... > ./ylwrap: line 175: yacc: command not found > make: *** [sieve/sieve.c] Error 127 > > > I made a patch (configure.ac.patch, attached) to check for yacc/lex > programs inside the Sieve option block as it's the only component (if > I'm not wrong) that requires yacc/lex. > > > Then I went ahead and also changed the following (I've already > mentioned both suggestions to Ken/list but received no response): > > Changed the behavior when no ICU4C lib is detected: from warning to > error as its absence causes an error during make stage, and other > similar required packages are reported inside configure with an error > message. > > Changed the type of message when com_err is absent or incomplete > (lacking compile_et): from warning to notice, as this is the most > common situation I believe (no or incomplete com_err) and it doesn't > affect in anything the result of the build process (its sources are > shipped with Cyrus sources and used when needed). > > > Inside this patch there is also a fix for a typo in a comment (says > "Build in *Sieve* support?" instead of "Build in *Squat* support?") > introduced with my previous patch for --disable-squat option fix. This > typo doesn't affect anything (it's just a comment for config.h.in), > but seizing the opportunity of fixing Makefile.am, here it is. > > > With all this applied, I see no warning/errors for the entire build > process when all needed packages are installed, and when not, > configure notifies me about all of them, not allowing to continue to > the make stage until all requirements are satisfied. Please apply the > fixes as you see fit. > > > Regards, > Anatoli > > > -----Original Message----- From: Cyrus-devel [mailto:cyrus-devel- > bounces+me=anatoli...@lists.andrew.cmu.edu] On Behalf Of ellie timoney > via Cyrus-devel Sent: Sunday, May 15, 2016 23:28 To: cyrus- > de...@lists.andrew.cmu.edu Subject: Re: v3.0 > > Hi Anatoli, > > Thanks for sending through the config.log and make output. > > For what it's worth, if you configure with --enable-silent-rules, the > make output will be a lot less noisy (and a lot easier to read). > > > Ken, the 2 errors that don't happen to you only happen to me when > > compiling > > with --enable-backup. I've tested the beta2 tarball as well as the > > current > > git, both errors still happen to me on 2 different machines. > > Without the > > --enable-backup option these errors don't occur. > > As I mentioned in private email, I don't get these errors either > (Debian). > > Poking around on Google, I see a few other projects experiencing the > "DSO missing from command line" error for Ubuntu builds. It > looks like > maybe Ubuntu's linker setup is more sensitive to correct order of > dependencies? > > I've attached an experimental patch which I think corrects the link > dependency order for the backup tools. Can you try it out? The patch > is against 3.0.0beta2 but should also apply cleanly against latest git > master. This should hopefully resolve the "DSO missing from command > line" errors when building with --enable-backup. Let me know how it > goes. > > > Warning (mostly harmless): No library found for -lcrypto > > (same for -lssl) > > I dug around a bit and I think these come from the perl module build > system. I'm not sure what they mean exactly, might be a red herring. > Your config.log shows it's correctly detecting OpenSSL, anyway. > > > I found another issue with the dependencies: the build requires > > yacc/lex > > when building sieve, though in the Release Notes there is the > > following > > text: Replaced the et (error table) libary with a version that > > doesn't > > require lex or yacc. Remove the lex/yacc checking from Configure. > > That release note item is ancient (version 1.4!) and I doubt it's of > much relevance now. Your config.log says: > > > configure:19090: checking for bison > > configure:19106: found /usr/bin/bison > > configure:19117: result: bison -y > > configure:19133: checking for flex > > configure:19149: found /usr/bin/flex > > configure:19160: result: flex > > so configure is looking for, and finding, bison (supercedes yacc) and > flex (supercedes lex). > > What problem are you seeing here? > > Cheers, > > ellie > > On Fri, May 13, 2016, at 05:15 PM, Anatoli via Cyrus-devel wrote: > > Hi all, > > > > Ken, the 2 errors that don't happen to you only happen to me when > > compiling > > with --enable-backup. I've tested the beta2 tarball as well as the > > current > > git, both errors still happen to me on 2 different machines. > > Without the > > --enable-backup option these errors don't occur. > > > > Ellie, I'll send the config.log and the resulting make output to > > you and > > Ken > > directly. > > > > > > I can confirm that the latest master from the dev git has the > > patch for > > --disable-squat correctly working and the ICU checking in > > configure is > > present now too, but maybe it should be converted from a > > warning to an > > error > > when with --enable-http? > > > > DKIM warning doesn't appearing anymore. It would be great to > > solve the > > "Parts of com_err distribuion were found, but not compile_et" > > warning > > too. > > > > I'm also getting: > > checking for BIO_accept in -lcrypto... yes > > checking for crypt... no > > checking for crypt in -lcrypt... yes > > Warning (mostly harmless): No library found for -lcrypto > > (same for -lssl) > > > > I found another issue with the dependencies: the build requires > > yacc/lex > > when building sieve, though in the Release Notes there is the > > following > > text: Replaced the et (error table) libary with a version that > > doesn't > > require lex or yacc. Remove the lex/yacc checking from Configure. > > > > I guess the lex/yacc checking should be placed in the configure > > again, > > inside the enable-sieve block. > > > > Regards, > > Anatoli > > > > > > -----Original Message----- > > From: Ken Murchison [mailto:mu...@andrew.cmu.edu] > > Sent: Wednesday, May 11, 2016 15:44 > > To: Anatoli; cyrus-devel@lists.andrew.cmu.edu > > Subject: Re: v3.0 > > > > > > > > On 05/09/2016 02:57 AM, Anatoli via Cyrus-devel wrote: > > > Hi all, > > > > > > I'm testing v3.0.0 beta2. Here goes the feedback, this time for > > > the build > > > process. > > > > > > 1. --disable-squat option in configure has no effect. Please see > > > attached > > a > > > patch (configure.ac.patch). > > > > Patch applied. > > > > > > > 2. Without icu-dev package make fails with: > > > > > > unicode/ucal.h No such file or directory > > > > > > It somehow depends on libical, i.e. it looks like icu-dev > > > should be > > > installed before building libical. In any case it should be > > > detected in > > > configure to avoid build errors. > > > > A check for ICU4C has been added to configure.ac > > > > > > > 3. With --enable-backup make fails with: > > > > > > /usr/bin/ld: backup/.libs/libcyrus_backup.a(lcb_append.o): > > > undefined > > > reference to symbol 'SHA1_Update@@OPENSSL_1.0.0' > > > //usr/lib64/libcrypto.so: error adding symbols: DSO missing from > > > command > > > line > > > > > > Problem: missing -lcrypto for libcyrus_backup.a > > > > > > /usr/bin/ld: imap/.libs/libcyrus_imap.so: undefined reference to > > > symbol > > > 'cyrusdb_fetch' > > > lib/.libs/libcyrus.so: error adding symbols: DSO missing from > > > command line > > > > > > Problem: missing lib/libcyrus.la for cyr_backup. > > > > > > Please see attached a patch for both errors (Makefile.am.patch). > > > > > > > > > With these fixes build completes without errors. > > > > I'm not seeing this problem compiling on my machine. I wonder why? > > > > > > > Then I got this new warning: > > > > > > configure: WARNING: Your version of OpenDKIM can not support > > > iSchedule. > > > Consider upgrading to OpenDKIM >= 2.7.0 > > > > > > I don't have DKIM on the box where Cyrus runs, in 2.5.7 there > > > was no > > warning > > > about DKIM. Haven't investigated the details yet. DKIM is part > > > of the > > > iSchedule draft, but is it required? And should it be OpenDKIM > > > only or any > > > other DKIM package works too? > > > > I have commented out the check for DKIM since its not needed for > > anything that sites will deploy any time soon. > > > > > > > And then there is an old warning (also happens in 2.5.7): > > > > > > configure: WARNING: Parts of com_err distribuion were found, > > > but not > > > compile_et. > > > configure: WARNING: Will build com_err from included sources. > > > > > > What should be installed/changed to avoid it? Or maybe the warning > > > itself > > > should be converted to a notice as compile_et is shipped with cyrus- > > > imap > > and > > > everything works as expected? > > > > > > Regards, > > > Anatoli > > > > > > > > > -----Original Message----- > > > From: Anatoli [mailto:m...@anatoli.ws] > > > Sent: Tuesday, April 19, 2016 03:39 > > > To: cyrus-devel@lists.andrew.cmu.edu > > > Subject: RE: v3.0 > > > > > > Hi Ellie, > > > > > > Thanks for the link! This is the information I was looking for. > > > Great, > > > there's a new beta! I'll test it these days and if it behaves > > > reasonably > > > well, I'll try to deploy it in a small production environment > > > where the > > > users are OK being beta-testers. I'll post here any issues found. > > > > > > With respect to the specialuse flags issue, it's not possible to > > > set these > > > flags from cyradm in the latest release (2.5.7). I'll check if > > > it's fixed > > in > > > the 3.0 beta. > > > > > > I have a feature suggestion with respect to these flags. I suppose > > > that > > most > > > of the deployments use these flags on some folders from the > > > autocreate_inbox_folders list. So, instead of writing scripts > > > that set > > these > > > flags somehow, what if it would be possible to specify the > > > specialuse flag > > > for each autocreate folder as an optional param (with some > > > (invalid for > > > folder names) char (e.g. ':') as the delimiter)? > > > > > > Something like this: > > > autocreate_inbox_folders: > > > Sent:Sent|Trash:trash|Drafts:DRAFTS|Spam:Junk|OtherFolder1|OtherF- > > > older2 > > > > > > The format would be defines as: > > > folder[:<specialuse_flag>][|folder[:<specialuse_flag>]]... and > > > <specialuse_flag> would be one of the options from the RFC6154 > > > (section > > 2), > > > interpreted case-insensitively. > > > > > > So when the user logs in for the first time, he/she has all the > > > folders > > > created with the necessary flags. IMO, a significant > > > simplification of the > > > mbox creation process. > > > > > > I haven't analyzed the code for this feature yet, but it should be > > > quite > > > simple to implement. Just parse the optional params, check if the > > > value is > > > in a predefined array and after creation of the folder, set the > > > requested > > > flag. Every flag could be validated for uniqueness or it may be > > > left up to > > > the Cyrus administrator to decide and specify the correct values > > > (I would > > > prefer the later, the RFC explicitly says it's up to each server > > > implementation (section 3)). > > > > > > Regards, > > > Anatoli > > > > > > -----Original Message----- > > > From: Cyrus-devel > > > [mailto:cyrus-devel-bounces+me=anatoli...@lists.andrew.cmu.edu] On > > > Behalf > > Of > > > ellie timoney via Cyrus-devel > > > Sent: Sunday, April 17, 2016 22:37 > > > To: cyrus-devel@lists.andrew.cmu.edu > > > Subject: Re: v3.0 > > > > > > Hi Anatoli, > > > > > >> I'm quite interested in this release and I'd > > >> like to help with testing, simple problems investigation and > > >> fixing, and > > >> similar tasks, > > > That would be greatly appreciated :) > > > > > >> but at least some insight to the current state of the > > >> master is needed for that, i.e. how close it is to an RC, new > > >> functionality expected to work, functionality/configuration > > >> changes > > >> compared to v2.5.7, known limitations, etc. > > > Have you looked at the 3.0.0-beta2 that was released last > > > week? Its > > > release notes compare it against the 2.5 series: > > > http://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html > > > > > >> I see the T232 by Ellie has 4 issues that are apparently stopping > > >> this > > >> release. Are they the only remaining issues for production-ready > > >> state? > > > These are just the very narrow intersection of a) what I'm > > > aware of, > > > that has b) been logged at all, and c) is logged in phabricator > > > where I > > > can mark it as blocking (rather than in bugzilla, the mailing > > > list, > > > private email, etc). > > > > > >> The roadmap > > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap) > > >> says > > >> nothing about v3.0 and looks a little outdated. > > > This page looks like a direct import of the page from the old > > > website. > > > I'm not sure quite how old it is, but given it's referencing > > > "2.6" as > > > future, that suggests that it's over a year old... > > > > > >> I'm personally interested in resolving the "specialuse flags not > > >> working > > >> from cyradm" (T199, 198, 191, 121; looks like still pending) > > >> issue > > > Are these still issues? Or are they stale tasks that have been > > > fixed > > > but not closed? We've had a number of cyradm metadata patches > > > contributed by a few different people over the last year, so it > > > seems > > > probable that at least some are fixed but the assignees don't > > > know. > > > > > >> Also I made a raw chroot patch for 2.5.7, > > >> I'd like to polish and submit it for review and inclusion in > > >> v3.0. > > > That'd be great! > > > > > >> Should I write to someone in particular to discuss the subject? > > > Everyone working on Cyrus 3.0 is active on this list, so this is > > > probably the best place for it. > > > > > > You're also welcome to join our conference calls, they happen at > > > 11am > > > UTC most Mondays on Google Hangouts. Probably the easiest way to > > > join > > > is to come into #cyrus on Freenode IRC at the meeting time and > > > ask for > > > the hangouts link, because it changes occasionally. > > > > > > Cheers, > > > > > > ellie > > > > > > On Sat, Apr 16, 2016, at 04:41 AM, Anatoli via Cyrus-devel wrote: > > >> Hi folks, > > >> > > >> Sorry for bothering you again with subj, I haven't received any > > >> answer to > > >> the previous mails about it. I'm quite interested in this release > > >> and I'd > > >> like to help with testing, simple problems investigation and > > >> fixing, and > > >> similar tasks, but at least some insight to the current state of > > >> the > > >> master is needed for that, i.e. how close it is to an RC, new > > >> functionality expected to work, functionality/configuration > > >> changes > > >> compared to v2.5.7, known limitations, etc. > > >> > > >> I see the T232 by Ellie has 4 issues that are apparently stopping > > >> this > > >> release. Are they the only remaining issues for production-ready > > >> state? > > >> The roadmap > > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap) > > >> says > > >> nothing about v3.0 and looks a little outdated. > > >> > > >> I'm personally interested in resolving the "specialuse flags not > > >> working > > >> from cyradm" (T199, 198, 191, 121; looks like still pending) > > >> issue and in > > >> XAPPLEPUSHSERVICE feature, but I know there are a lot of other > > >> improvements and new libraries support (like LibiCal2.0) that are > > >> worth > > >> the effort releasing it ASAP. Also I made a raw chroot patch for > > >> 2.5.7, > > >> I'd like to polish and submit it for review and inclusion in > > >> v3.0. > > >> > > >> Should I write to someone in particular to discuss the subject? > > >> > > >> Regards, > > >> Anatoli > > >> > > >> > > > > -- > > Kenneth Murchison > > Principal Systems Software Engineer > > Carnegie Mellon University > > > Email had 2 attachments: > * Makefile.am.patch 1k (text/plain) > * configure.ac.patch 3k (text/plain)
