>> I made a patch (configure.ac.patch, attached) >> [...] >> Please apply the fixes as you see fit. > > Thanks, those look good to me. I'll push it up, but I'll split it > into separate commits for each piece (sieve/RSCALE/com_err) for > historical clarity. I'll also fix the "distribuion" typo while I'm in > there, that's been annoying me for ages. :P The configure.ac changes are now (mostly) on master. I left out one of the WARN=>NOTICE changes for com_err because it looks like its intent is to alert the user that we're going against their wishes, and using our com_err when they explicitly asked us to not use any com_err at all. That seems like it should still be at least a warning, if not an error. Though, I did uncover a (largely inconsequential) bug, whereby this warning doesn't fire when it ought to anyway: https://git.cyrus.foundation/T240 On Tue, May 17, 2016, at 01:16 PM, ellie timoney via Cyrus-devel wrote: > Hi Anatoli, > >> Ellie, the patch that you've sent for the backup linking problem >> doesn't solve the issue > > Does anything different happen, or is it exactly the same as without > it? Can you provide output? I'm pretty sure my patch is on the right > track, but there's almost certainly a detail I've missed since I can't > reproduce it myself. > > The other thing I wanted to ask was if you could run make with "-k". > I have a hunch the other backup tools will currently fail to link for > you in the exact same way, but if make stops at the first failure then > we don't get to see that, nor whether they fail or succeed for some > other reason (which might provide a clue). > >> There is a detailed explanation of the issue here: >> https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries >> and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition. > > Thanks for tracking down those links, I'll look through them. > >> I've already proposed to Ken a slightly different patch than yours >> (Makefile.am.patch, attached to this email is its version with >> adaptations to apply smoothly to the latest master commit 9c8e8ea) >> that works well under these rules. >> >> The changes are: to add SSL_LIBS to the list of the libs for >> libcyrus_backup (currently there are none) and for the second error >> (that was mentioned in the initial mail about these problems) to add >> lib/libcyrus.la to the list of the libs for cyr_backup. >> > > Yep, I saw that patch. The reasons my patch is different are that: > > * libcyrus_backup.la is a no-install library. At present, I expect > anything linking against it to also link against anything it needs > (hence not having any LDADD of its own). It sounds like this needs to > change because of the SHA1_Update call in the library. > * libcyrus.la is in $LD_BASIC_ADD, which is in $LD_UTILITY_ADD, and > thus is already on the list for cyr_backup and the other backup tools > (but clearly in the wrong order) > * I want, as much as possible, to avoid having to manually specify > lists of library dependencies for build targets. (That gets > unmaintainable very quickly) > > So I'm looking for a way to get the libraries specified in the correct > order, portably, and with minimal hand maintenance of dependency > lists. Given that the existing system works for everything but the > backup tools (with their internal API library), and given that > literally nobody but me uses the backup tools at the moment (afaik), > there's an opportunity here to integrate them correctly, if a way can > be found to do so. > > If we just manually add them to the lists, then next time something > like this shows up we'll still have no idea how to fix it properly > >> I made a patch (configure.ac.patch, attached) >> [...] >> Please apply the fixes as you see fit. > > Thanks, those look good to me. I'll push it up, but I'll split it > into separate commits for each piece (sieve/RSCALE/com_err) for > historical clarity. I'll also fix the "distribuion" typo while I'm in > there, that's been annoying me for ages. :P > > Cheers, > > ellie > > On Tue, May 17, 2016, at 10:21 AM, Anatoli via Cyrus-devel wrote: >> Hi All, >> >> Ellie, the patch that you've sent for the backup linking problem >> doesn't solve the issue, but you're right, I've just dug deeper: it's >> a matter of order of the library inclusion options and it's because >> in Ubuntu starting from 11.04 (and some versions of Debian too) the >> gcc invokes ld with --as-needed flag that makes the order of >> libraries quite important. There is a detailed explanation of the >> issue here: >> https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries >> and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition. >> >> Also it looks like newer versions of gcc/ld (mine is 4.8.4/2.24) >> treat the order of includes much more strictly than previous versions >> on most platforms. >> >> In a nutshell, a binary/library (A) which calls an external function >> defined in another library (B) should appear *before* the library (B) >> containing the function. The reason is that gcc/ld with --as-needed >> flag optimizes symbols that aren't referenced, so if it sees library >> B first, and no one has referenced it at that point then it won't >> link in anything from it at all. This is what happens with -lcrypto >> going before libcyrus_backup.a: the SHA1_Update function is only used >> inside libcyrus_backup.a but the linker has already stripped it from >> -lcrypto. If we include -lcrypto again *after* libcyrus_backup.a, >> everything builds fine. >> >> >> I've already proposed to Ken a slightly different patch than yours >> (Makefile.am.patch, attached to this email is its version with >> adaptations to apply smoothly to the latest master commit 9c8e8ea) >> that works well under these rules. >> >> The changes are: to add SSL_LIBS to the list of the libs for >> libcyrus_backup (currently there are none) and for the second error >> (that was mentioned in the initial mail about these problems) to add >> lib/libcyrus.la to the list of the libs for cyr_backup. >> >> So, instead of expecting the include libs, that are required for >> certain parts of the binary/lib we're building, to make random >> appearance in the gcc line (in sometimes incorrect order under the >> strict/as-needed rules), we add in the Makefile.am the needed libs >> for each and every lib/block of code we use to compose the final >> output. As a result, at the linking stage they appear in the >> correct order. >> >> >> >> >> Warning (mostly harmless): No library found for -lcrypto >> >> Indeed, it's a Makefile.PL warning as my perl installation LD path is >> not complete (doesn't include /usr/lib64 where crypto/ssl libs are). >> It should be fixed locally in my perl deployment. In any case it's >> indeed harmless as gcc/ld knows about this path and this is where >> it's used. >> >> >> >> >> The yacc/lex dependency is actually not checked. Better to say >> configure tries to determine what yacc/lex progs are installed, but >> if it founds none, it continues as if everything's OK. The config.log >> I've sent you was for the build with the packages installed >> (otherwise you can't proceed with the make stage). Without the >> packages, in the config.log I get: >> >> configure:19090: checking for bison >> configure:19120: result: no >> configure:19090: checking for byacc >> configure:19120: result: no >> configure:19133: checking for flex >> configure:19163: result: no >> configure:19133: checking for lex >> configure:19163: result: no >> >> but nothing more. So, configure reports no warnings/errors with >> respect to the missing tools and then with make I get: >> >> ./ylwrap: line 175: yacc: command not found >> make: *** [sieve/addr.c] Error 127 >> make: *** Waiting for unfinished jobs.... >> ./ylwrap: line 175: yacc: command not found >> make: *** [sieve/sieve.c] Error 127 >> >> >> I made a patch (configure.ac.patch, attached) to check for yacc/lex >> programs inside the Sieve option block as it's the only component (if >> I'm not wrong) that requires yacc/lex. >> >> >> Then I went ahead and also changed the following (I've already >> mentioned both suggestions to Ken/list but received no response): >> >> Changed the behavior when no ICU4C lib is detected: from warning to >> error as its absence causes an error during make stage, and other >> similar required packages are reported inside configure with an error >> message. >> >> Changed the type of message when com_err is absent or incomplete >> (lacking compile_et): from warning to notice, as this is the most >> common situation I believe (no or incomplete com_err) and it doesn't >> affect in anything the result of the build process (its sources are >> shipped with Cyrus sources and used when needed). >> >> >> Inside this patch there is also a fix for a typo in a comment (says >> "Build in *Sieve* support?" instead of "Build in *Squat* support?") >> introduced with my previous patch for --disable-squat option fix. >> This typo doesn't affect anything (it's just a comment for >> config.h.in), but seizing the opportunity of fixing Makefile.am, >> here it is. >> >> >> With all this applied, I see no warning/errors for the entire build >> process when all needed packages are installed, and when not, >> configure notifies me about all of them, not allowing to continue to >> the make stage until all requirements are satisfied. Please apply the >> fixes as you see fit. >> >> >> Regards, >> Anatoli >> >> >> -----Original Message----- From: Cyrus-devel [mailto:cyrus-devel- >> bounces+me=anatoli...@lists.andrew.cmu.edu] On Behalf Of ellie >> timoney via Cyrus-devel Sent: Sunday, May 15, 2016 23:28 To: cyrus- >> de...@lists.andrew.cmu.edu Subject: Re: v3.0 >> >> Hi Anatoli, >> >> Thanks for sending through the config.log and make output. >> >> For what it's worth, if you configure with --enable-silent-rules, the >> make output will be a lot less noisy (and a lot easier to read). >> >> > Ken, the 2 errors that don't happen to you only happen to me when >> > compiling >> > with --enable-backup. I've tested the beta2 tarball as well as the >> > current >> > git, both errors still happen to me on 2 different machines. >> > Without the >> > --enable-backup option these errors don't occur. >> >> As I mentioned in private email, I don't get these errors either >> (Debian). >> >> Poking around on Google, I see a few other projects experiencing the >> "DSO missing from command line" error for Ubuntu builds. It >> looks like >> maybe Ubuntu's linker setup is more sensitive to correct order of >> dependencies? >> >> I've attached an experimental patch which I think corrects the link >> dependency order for the backup tools. Can you try it out? >> The patch >> is against 3.0.0beta2 but should also apply cleanly against >> latest git >> master. This should hopefully resolve the "DSO missing from command >> line" errors when building with --enable-backup. Let me know how it >> goes. >> >> > Warning (mostly harmless): No library found for -lcrypto >> > (same for -lssl) >> >> I dug around a bit and I think these come from the perl module build >> system. I'm not sure what they mean exactly, might be a red herring. >> Your config.log shows it's correctly detecting OpenSSL, anyway. >> >> > I found another issue with the dependencies: the build requires >> > yacc/lex >> > when building sieve, though in the Release Notes there is the >> > following >> > text: Replaced the et (error table) libary with a version that >> > doesn't >> > require lex or yacc. Remove the lex/yacc checking from Configure. >> >> That release note item is ancient (version 1.4!) and I doubt it's of >> much relevance now. Your config.log says: >> >> > configure:19090: checking for bison >> > configure:19106: found /usr/bin/bison >> > configure:19117: result: bison -y >> > configure:19133: checking for flex >> > configure:19149: found /usr/bin/flex >> > configure:19160: result: flex >> >> so configure is looking for, and finding, bison (supercedes yacc) and >> flex (supercedes lex). >> >> What problem are you seeing here? >> >> Cheers, >> >> ellie >> >> On Fri, May 13, 2016, at 05:15 PM, Anatoli via Cyrus-devel wrote: >> > Hi all, >> > >> > Ken, the 2 errors that don't happen to you only happen to me when >> > compiling >> > with --enable-backup. I've tested the beta2 tarball as well as the >> > current >> > git, both errors still happen to me on 2 different machines. >> > Without the >> > --enable-backup option these errors don't occur. >> > >> > Ellie, I'll send the config.log and the resulting make output to >> > you and >> > Ken >> > directly. >> > >> > >> > I can confirm that the latest master from the dev git has the >> > patch for >> > --disable-squat correctly working and the ICU checking in >> > configure is >> > present now too, but maybe it should be converted from a warning >> > to an >> > error >> > when with --enable-http? >> > >> > DKIM warning doesn't appearing anymore. It would be great to >> > solve the >> > "Parts of com_err distribuion were found, but not compile_et" >> > warning >> > too. >> > >> > I'm also getting: >> > checking for BIO_accept in -lcrypto... yes >> > checking for crypt... no >> > checking for crypt in -lcrypt... yes >> > Warning (mostly harmless): No library found for -lcrypto >> > (same for -lssl) >> > >> > I found another issue with the dependencies: the build requires >> > yacc/lex >> > when building sieve, though in the Release Notes there is the >> > following >> > text: Replaced the et (error table) libary with a version that >> > doesn't >> > require lex or yacc. Remove the lex/yacc checking from Configure. >> > >> > I guess the lex/yacc checking should be placed in the configure >> > again, >> > inside the enable-sieve block. >> > >> > Regards, >> > Anatoli >> > >> > >> > -----Original Message----- >> > From: Ken Murchison [mailto:mu...@andrew.cmu.edu] >> > Sent: Wednesday, May 11, 2016 15:44 >> > To: Anatoli; cyrus-devel@lists.andrew.cmu.edu >> > Subject: Re: v3.0 >> > >> > >> > >> > On 05/09/2016 02:57 AM, Anatoli via Cyrus-devel wrote: >> > > Hi all, >> > > >> > > I'm testing v3.0.0 beta2. Here goes the feedback, this time for >> > > the build >> > > process. >> > > >> > > 1. --disable-squat option in configure has no effect. Please see >> > > attached >> > a >> > > patch (configure.ac.patch). >> > >> > Patch applied. >> > >> > >> > > 2. Without icu-dev package make fails with: >> > > >> > > unicode/ucal.h No such file or directory >> > > >> > > It somehow depends on libical, i.e. it looks like icu-dev >> > > should be >> > > installed before building libical. In any case it should be >> > > detected in >> > > configure to avoid build errors. >> > >> > A check for ICU4C has been added to configure.ac >> > >> > >> > > 3. With --enable-backup make fails with: >> > > >> > > /usr/bin/ld: backup/.libs/libcyrus_backup.a(lcb_append.o): >> > > undefined >> > > reference to symbol 'SHA1_Update@@OPENSSL_1.0.0' >> > > //usr/lib64/libcrypto.so: error adding symbols: DSO missing from >> > > command >> > > line >> > > >> > > Problem: missing -lcrypto for libcyrus_backup.a >> > > >> > > /usr/bin/ld: imap/.libs/libcyrus_imap.so: undefined reference to >> > > symbol >> > > 'cyrusdb_fetch' >> > > lib/.libs/libcyrus.so: error adding symbols: DSO missing from >> > > command line >> > > >> > > Problem: missing lib/libcyrus.la for cyr_backup. >> > > >> > > Please see attached a patch for both errors (Makefile.am.patch). >> > > >> > > >> > > With these fixes build completes without errors. >> > >> > I'm not seeing this problem compiling on my machine. I wonder why? >> > >> > >> > > Then I got this new warning: >> > > >> > > configure: WARNING: Your version of OpenDKIM can not support >> > > iSchedule. >> > > Consider upgrading to OpenDKIM >= 2.7.0 >> > > >> > > I don't have DKIM on the box where Cyrus runs, in 2.5.7 there >> > > was no >> > warning >> > > about DKIM. Haven't investigated the details yet. DKIM is part of >> > > the >> > > iSchedule draft, but is it required? And should it be OpenDKIM >> > > only or any >> > > other DKIM package works too? >> > >> > I have commented out the check for DKIM since its not needed for >> > anything that sites will deploy any time soon. >> > >> > >> > > And then there is an old warning (also happens in 2.5.7): >> > > >> > > configure: WARNING: Parts of com_err distribuion were found, but >> > > not >> > > compile_et. >> > > configure: WARNING: Will build com_err from included sources. >> > > >> > > What should be installed/changed to avoid it? Or maybe the >> > > warning itself >> > > should be converted to a notice as compile_et is shipped with cyrus- >> > > imap >> > and >> > > everything works as expected? >> > > >> > > Regards, >> > > Anatoli >> > > >> > > >> > > -----Original Message----- >> > > From: Anatoli [mailto:m...@anatoli.ws] >> > > Sent: Tuesday, April 19, 2016 03:39 >> > > To: cyrus-devel@lists.andrew.cmu.edu >> > > Subject: RE: v3.0 >> > > >> > > Hi Ellie, >> > > >> > > Thanks for the link! This is the information I was looking for. >> > > Great, >> > > there's a new beta! I'll test it these days and if it behaves >> > > reasonably >> > > well, I'll try to deploy it in a small production environment >> > > where the >> > > users are OK being beta-testers. I'll post here any issues found. >> > > >> > > With respect to the specialuse flags issue, it's not possible to >> > > set these >> > > flags from cyradm in the latest release (2.5.7). I'll check if >> > > it's fixed >> > in >> > > the 3.0 beta. >> > > >> > > I have a feature suggestion with respect to these flags. I >> > > suppose that >> > most >> > > of the deployments use these flags on some folders from the >> > > autocreate_inbox_folders list. So, instead of writing scripts >> > > that set >> > these >> > > flags somehow, what if it would be possible to specify the >> > > specialuse flag >> > > for each autocreate folder as an optional param (with some >> > > (invalid for >> > > folder names) char (e.g. ':') as the delimiter)? >> > > >> > > Something like this: >> > > autocreate_inbox_folders: >> > > Sent:Sent|Trash:trash|Drafts:DRAFTS|Spam:Junk|OtherFolder1|Other- >> > > Folder2 >> > > >> > > The format would be defines as: >> > > folder[:<specialuse_flag>][|folder[:<specialuse_flag>]]... and >> > > <specialuse_flag> would be one of the options from the RFC6154 >> > > (section >> > 2), >> > > interpreted case-insensitively. >> > > >> > > So when the user logs in for the first time, he/she has all the >> > > folders >> > > created with the necessary flags. IMO, a significant >> > > simplification of the >> > > mbox creation process. >> > > >> > > I haven't analyzed the code for this feature yet, but it should >> > > be quite >> > > simple to implement. Just parse the optional params, check if the >> > > value is >> > > in a predefined array and after creation of the folder, set the >> > > requested >> > > flag. Every flag could be validated for uniqueness or it may be >> > > left up to >> > > the Cyrus administrator to decide and specify the correct values >> > > (I would >> > > prefer the later, the RFC explicitly says it's up to each server >> > > implementation (section 3)). >> > > >> > > Regards, >> > > Anatoli >> > > >> > > -----Original Message----- >> > > From: Cyrus-devel >> > > [mailto:cyrus-devel-bounces+me=anatoli...@lists.andrew.cmu.edu] >> > > On Behalf >> > Of >> > > ellie timoney via Cyrus-devel >> > > Sent: Sunday, April 17, 2016 22:37 >> > > To: cyrus-devel@lists.andrew.cmu.edu >> > > Subject: Re: v3.0 >> > > >> > > Hi Anatoli, >> > > >> > >> I'm quite interested in this release and I'd >> > >> like to help with testing, simple problems investigation and >> > >> fixing, and >> > >> similar tasks, >> > > That would be greatly appreciated :) >> > > >> > >> but at least some insight to the current state of the >> > >> master is needed for that, i.e. how close it is to an RC, new >> > >> functionality expected to work, functionality/configuration >> > >> changes >> > >> compared to v2.5.7, known limitations, etc. >> > > Have you looked at the 3.0.0-beta2 that was released last week? >> > > Its >> > > release notes compare it against the 2.5 series: >> > > http://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html >> > > >> > >> I see the T232 by Ellie has 4 issues that are apparently >> > >> stopping this >> > >> release. Are they the only remaining issues for production-ready >> > >> state? >> > > These are just the very narrow intersection of a) what I'm aware >> > > of, >> > > that has b) been logged at all, and c) is logged in phabricator >> > > where I >> > > can mark it as blocking (rather than in bugzilla, the mailing >> > > list, >> > > private email, etc). >> > > >> > >> The roadmap >> > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap) >> > >> says >> > >> nothing about v3.0 and looks a little outdated. >> > > This page looks like a direct import of the page from the old >> > > website. >> > > I'm not sure quite how old it is, but given it's referencing >> > > "2.6" as >> > > future, that suggests that it's over a year old... >> > > >> > >> I'm personally interested in resolving the "specialuse flags not >> > >> working >> > >> from cyradm" (T199, 198, 191, 121; looks like still pending) >> > >> issue >> > > Are these still issues? Or are they stale tasks that have been >> > > fixed >> > > but not closed? We've had a number of cyradm metadata patches >> > > contributed by a few different people over the last year, so it >> > > seems >> > > probable that at least some are fixed but the assignees don't >> > > know. >> > > >> > >> Also I made a raw chroot patch for 2.5.7, >> > >> I'd like to polish and submit it for review and inclusion in >> > >> v3.0. >> > > That'd be great! >> > > >> > >> Should I write to someone in particular to discuss the subject? >> > > Everyone working on Cyrus 3.0 is active on this list, so this is >> > > probably the best place for it. >> > > >> > > You're also welcome to join our conference calls, they happen at >> > > 11am >> > > UTC most Mondays on Google Hangouts. Probably the easiest way to >> > > join >> > > is to come into #cyrus on Freenode IRC at the meeting time and >> > > ask for >> > > the hangouts link, because it changes occasionally. >> > > >> > > Cheers, >> > > >> > > ellie >> > > >> > > On Sat, Apr 16, 2016, at 04:41 AM, Anatoli via Cyrus-devel wrote: >> > >> Hi folks, >> > >> >> > >> Sorry for bothering you again with subj, I haven't received any >> > >> answer to >> > >> the previous mails about it. I'm quite interested in this >> > >> release and I'd >> > >> like to help with testing, simple problems investigation and >> > >> fixing, and >> > >> similar tasks, but at least some insight to the current state of >> > >> the >> > >> master is needed for that, i.e. how close it is to an RC, new >> > >> functionality expected to work, functionality/configuration >> > >> changes >> > >> compared to v2.5.7, known limitations, etc. >> > >> >> > >> I see the T232 by Ellie has 4 issues that are apparently >> > >> stopping this >> > >> release. Are they the only remaining issues for production-ready >> > >> state? >> > >> The roadmap >> > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap) >> > >> says >> > >> nothing about v3.0 and looks a little outdated. >> > >> >> > >> I'm personally interested in resolving the "specialuse flags not >> > >> working >> > >> from cyradm" (T199, 198, 191, 121; looks like still pending) >> > >> issue and in >> > >> XAPPLEPUSHSERVICE feature, but I know there are a lot of other >> > >> improvements and new libraries support (like LibiCal2.0) that >> > >> are worth >> > >> the effort releasing it ASAP. Also I made a raw chroot patch for >> > >> 2.5.7, >> > >> I'd like to polish and submit it for review and inclusion in >> > >> v3.0. >> > >> >> > >> Should I write to someone in particular to discuss the subject? >> > >> >> > >> Regards, >> > >> Anatoli >> > >> >> > >> >> > >> > -- >> > Kenneth Murchison >> > Principal Systems Software Engineer >> > Carnegie Mellon University >> > >> Email had 2 attachments:
>> * Makefile.am.patch 1k (text/plain) >> * configure.ac.patch 3k (text/plain) >
