[PATCH] lib: make notmuch shared library install_name be full path on Mac OS X
The install_name of libnotmuch.dylib on Mac OS X is what is written into a program that links against it. If it is just the name of the shared library file, as opposed to the full path, the program won't be able to find it when it runs and will abort. Instead, the install_name should be the full path to the shared library (in its final installed location). An example on Mac OS X Mavericks (10.9.4) follows. We configure, build, and install notmuch, assuming the user has write access to /opt: === $ CFLAGS='-I/pkg/include' LDFLAGS='-L/pkg/lib' ./configure \ --prefix=/opt/notmuch-current --without-emacs $ make $ make install === We inspect the installed notmuch library using the otool system program: === $ otool -L /opt/notmuch-current/lib/libnotmuch.dylib /opt/notmuch-current/lib/libnotmuch.dylib: libnotmuch.3.dylib \ (compatibility version 3.1.0, current version 3.1.0) /pkg/lib/libgmime-2.4.2.dylib \ (compatibility version 7.0.0, current version 7.33.0) /pkg/lib/libgobject-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libglib-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libintl.8.dylib \ (compatibility version 10.0.0, current version 10.2.0) /pkg/lib/libtalloc.2.1.1.dylib \ (compatibility version 0.0.0, current version 0.0.0) /pkg/lib/libz.1.dylib \ (compatibility version 2.0.0, current version 2.2.0) /pkg/lib/libxapian.22.dylib \ (compatibility version 29.0.0, current version 29.4.0) /usr/lib/libc++.1.dylib \ (compatibility version 1.0.0, current version 120.0.0) /usr/lib/libSystem.B.dylib \ (compatibility version 1.0.0, current version 1197.1.1) === That's not good. The second line shows the install_name, and it's not a full path, so any program that links against it as a dependent library will not be able to find it. We try running the notmuch program (which is linked against the notmuch library), and, sure enough, it aborts: === $ /opt/notmuch-current/bin/notmuch --version dyld: Library not loaded: libnotmuch.3.dylib Referenced from: /opt/notmuch-current/bin/notmuch Reason: image not found Trace/BPT trap: 5 === After applying this commit to fix the problem and configuring, building, and installing again as above, the notmuch library's install_name is now the correct full path: === $ otool -L /opt/notmuch-current/lib/libnotmuch.dylib /opt/notmuch-current/lib/libnotmuch.dylib: /opt/notmuch-current/lib/libnotmuch.3.dylib \ (compatibility version 3.1.0, current version 3.1.0) /pkg/lib/libgmime-2.4.2.dylib \ (compatibility version 7.0.0, current version 7.33.0) /pkg/lib/libgobject-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libglib-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libintl.8.dylib \ (compatibility version 10.0.0, current version 10.2.0) /pkg/lib/libtalloc.2.1.1.dylib \ (compatibility version 0.0.0, current version 0.0.0) /pkg/lib/libz.1.dylib \ (compatibility version 2.0.0, current version 2.2.0) /pkg/lib/libxapian.22.dylib \ (compatibility version 29.0.0, current version 29.4.0) /usr/lib/libc++.1.dylib \ (compatibility version 1.0.0, current version 120.0.0) /usr/lib/libSystem.B.dylib \ (compatibility version 1.0.0, current version 1197.1.1) === And the notmuch program is able to find the notmuch library when run: === $ /opt/notmuch-current/bin/notmuch --version notmuch 0.18.1+84~g658a00e === References: * http://www.finkproject.org/doc/porting/porting.en.html#\ shared.build-lib * https://developer.apple.com/library/mac/documentation/DeveloperTools/\ Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html --- lib/Makefile.local | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Makefile.local b/lib/Makefile.local index c56cba9..c83f387 100644 --- a/lib/Makefile.local +++ b/lib/Makefile.local @@ -27,7 +27,7 @@ LIBRARY_SUFFIX = dylib LINKER_NAME = libnotmuch.$(LIBRARY_SUFFIX) SONAME = libnotmuch.$(LIBNOTMUCH_VERSION_MAJOR).$(LIBRARY_SUFFIX) LIBNAME = libnotmuch.$(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE).$(LIBRARY_SUFFIX) -LIBRARY_LINK_FLAG = -dynamiclib -install_name $(SONAME) -compatibility_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR) -current_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE) +LIBRARY_LINK_FLAG = -dynamiclib -install_name $(libdir)/$(SONAME) -compatibility_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR) -current_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE) else LIBRARY_SUFFIX = so LINKER_NAME = libnotmuch.$(LIBRARY_SUFFIX) -- 1.8.5.2 (Apple Git-48)
How to debug 'ignoring non-mail file' issues
Hi, I indexed my archive of emails from recent years with notmuch (about 10k messages so not much). I have quite a lot of messages 'notmuch new' ignores as non-mail files, about 1000 of them. They are not obviously malformed, meaning that the files certainly look like emails when opened in a text editor. I'd like to find out why these files are ignored, and if there is something I can do to fix them. Of course I'd like to have a complete database of my old emails, with nothing falling through the cracks like this. The vast majority of these ignored mails are not ignored after I transfer them with offlineimap to another computer. I can non-ignore these files probably by copying the renamed file back to the mail server, so this is fixable. Offlineimap shouldn't mess with the file's contents, so is there something that can cause notmuch to ignore a file based on its name? Looking at the rest of the ignored messages most of them seem to have very large attachments, but there are possibly others. There is only maybe 20 of these kinds of emails so I can try to fix them manually. Still, it would help if I knew what exactly caused notmuch to ignore the file. I understand most of the message parsing is done with gmime. Does gmime give any diagnostics on parse errors that could be used to give a reason for thinking a file is not mail? I understand that the list of non-mail files is stored in the notmuch database and the files are completely ignored from there on. This actually makes it harder to debug these kind of issues since the list of ignored mails is only visible on the first invocation of 'notmuch new', unless the files are moved around. Is there some way to extract the list of ignored files from the database for inspection? Maybe 'notmuch new' could have some kind of --unignore-non-mail switch that would reconsider previously ignored files. -- Perttu Luukko
How to debug 'ignoring non-mail file' issues
Perttu Luukko writes:
> I understand that the list of non-mail files is stored in the notmuch
> database and the files are completely ignored from there on. This
> actually makes it harder to debug these kind of issues since the list of
> ignored mails is only visible on the first invocation of 'notmuch new',
> unless the files are moved around. Is there some way to extract the list
> of ignored files from the database for inspection? Maybe 'notmuch new'
> could have some kind of --unignore-non-mail switch that would reconsider
> previously ignored files.
I _think_ it should suffice to do something like
find Maildir -type d -exec touch {} \;
to force a rescan
d
How to debug 'ignoring non-mail file' issues
Perttu Luukko writes: > The vast majority of these ignored mails are not ignored after I > transfer them with offlineimap to another computer. I can non-ignore > these files probably by copying the renamed file back to the mail > server, so this is fixable. Offlineimap shouldn't mess with the file's > contents, so is there something that can cause notmuch to ignore a file > based on its name? The most likely cause is that the files are mboxes, whether intentional or not. In particular if they start with a "From " (note the lack of :) and contain a second "From " at the beginning of a line later in the file. In this case something like sed can replace the initial "From " with "X-Envelope-From: ". I agree that the error message could be more informative in this case. d
How to debug 'ignoring non-mail file' issues
Hi, I indexed my archive of emails from recent years with notmuch (about 10k messages so not much). I have quite a lot of messages 'notmuch new' ignores as non-mail files, about 1000 of them. They are not obviously malformed, meaning that the files certainly look like emails when opened in a text editor. I'd like to find out why these files are ignored, and if there is something I can do to fix them. Of course I'd like to have a complete database of my old emails, with nothing falling through the cracks like this. The vast majority of these ignored mails are not ignored after I transfer them with offlineimap to another computer. I can non-ignore these files probably by copying the renamed file back to the mail server, so this is fixable. Offlineimap shouldn't mess with the file's contents, so is there something that can cause notmuch to ignore a file based on its name? Looking at the rest of the ignored messages most of them seem to have very large attachments, but there are possibly others. There is only maybe 20 of these kinds of emails so I can try to fix them manually. Still, it would help if I knew what exactly caused notmuch to ignore the file. I understand most of the message parsing is done with gmime. Does gmime give any diagnostics on parse errors that could be used to give a reason for thinking a file is not mail? I understand that the list of non-mail files is stored in the notmuch database and the files are completely ignored from there on. This actually makes it harder to debug these kind of issues since the list of ignored mails is only visible on the first invocation of 'notmuch new', unless the files are moved around. Is there some way to extract the list of ignored files from the database for inspection? Maybe 'notmuch new' could have some kind of --unignore-non-mail switch that would reconsider previously ignored files. -- Perttu Luukko ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: How to debug 'ignoring non-mail file' issues
Perttu Luukko perttu.luu...@iki.fi writes: The vast majority of these ignored mails are not ignored after I transfer them with offlineimap to another computer. I can non-ignore these files probably by copying the renamed file back to the mail server, so this is fixable. Offlineimap shouldn't mess with the file's contents, so is there something that can cause notmuch to ignore a file based on its name? The most likely cause is that the files are mboxes, whether intentional or not. In particular if they start with a From (note the lack of :) and contain a second From at the beginning of a line later in the file. In this case something like sed can replace the initial From with X-Envelope-From: . I agree that the error message could be more informative in this case. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: How to debug 'ignoring non-mail file' issues
Perttu Luukko perttu.luu...@iki.fi writes:
I understand that the list of non-mail files is stored in the notmuch
database and the files are completely ignored from there on. This
actually makes it harder to debug these kind of issues since the list of
ignored mails is only visible on the first invocation of 'notmuch new',
unless the files are moved around. Is there some way to extract the list
of ignored files from the database for inspection? Maybe 'notmuch new'
could have some kind of --unignore-non-mail switch that would reconsider
previously ignored files.
I _think_ it should suffice to do something like
find Maildir -type d -exec touch {} \;
to force a rescan
d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] lib: make notmuch shared library install_name be full path on Mac OS X
The install_name of libnotmuch.dylib on Mac OS X is what is written into a program that links against it. If it is just the name of the shared library file, as opposed to the full path, the program won't be able to find it when it runs and will abort. Instead, the install_name should be the full path to the shared library (in its final installed location). An example on Mac OS X Mavericks (10.9.4) follows. We configure, build, and install notmuch, assuming the user has write access to /opt: === $ CFLAGS='-I/pkg/include' LDFLAGS='-L/pkg/lib' ./configure \ --prefix=/opt/notmuch-current --without-emacs $ make $ make install === We inspect the installed notmuch library using the otool system program: === $ otool -L /opt/notmuch-current/lib/libnotmuch.dylib /opt/notmuch-current/lib/libnotmuch.dylib: libnotmuch.3.dylib \ (compatibility version 3.1.0, current version 3.1.0) /pkg/lib/libgmime-2.4.2.dylib \ (compatibility version 7.0.0, current version 7.33.0) /pkg/lib/libgobject-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libglib-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libintl.8.dylib \ (compatibility version 10.0.0, current version 10.2.0) /pkg/lib/libtalloc.2.1.1.dylib \ (compatibility version 0.0.0, current version 0.0.0) /pkg/lib/libz.1.dylib \ (compatibility version 2.0.0, current version 2.2.0) /pkg/lib/libxapian.22.dylib \ (compatibility version 29.0.0, current version 29.4.0) /usr/lib/libc++.1.dylib \ (compatibility version 1.0.0, current version 120.0.0) /usr/lib/libSystem.B.dylib \ (compatibility version 1.0.0, current version 1197.1.1) === That's not good. The second line shows the install_name, and it's not a full path, so any program that links against it as a dependent library will not be able to find it. We try running the notmuch program (which is linked against the notmuch library), and, sure enough, it aborts: === $ /opt/notmuch-current/bin/notmuch --version dyld: Library not loaded: libnotmuch.3.dylib Referenced from: /opt/notmuch-current/bin/notmuch Reason: image not found Trace/BPT trap: 5 === After applying this commit to fix the problem and configuring, building, and installing again as above, the notmuch library's install_name is now the correct full path: === $ otool -L /opt/notmuch-current/lib/libnotmuch.dylib /opt/notmuch-current/lib/libnotmuch.dylib: /opt/notmuch-current/lib/libnotmuch.3.dylib \ (compatibility version 3.1.0, current version 3.1.0) /pkg/lib/libgmime-2.4.2.dylib \ (compatibility version 7.0.0, current version 7.33.0) /pkg/lib/libgobject-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libglib-2.0.0.dylib \ (compatibility version 3801.0.0, current version 3801.2.0) /pkg/lib/libintl.8.dylib \ (compatibility version 10.0.0, current version 10.2.0) /pkg/lib/libtalloc.2.1.1.dylib \ (compatibility version 0.0.0, current version 0.0.0) /pkg/lib/libz.1.dylib \ (compatibility version 2.0.0, current version 2.2.0) /pkg/lib/libxapian.22.dylib \ (compatibility version 29.0.0, current version 29.4.0) /usr/lib/libc++.1.dylib \ (compatibility version 1.0.0, current version 120.0.0) /usr/lib/libSystem.B.dylib \ (compatibility version 1.0.0, current version 1197.1.1) === And the notmuch program is able to find the notmuch library when run: === $ /opt/notmuch-current/bin/notmuch --version notmuch 0.18.1+84~g658a00e === References: * http://www.finkproject.org/doc/porting/porting.en.html#\ shared.build-lib * https://developer.apple.com/library/mac/documentation/DeveloperTools/\ Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html --- lib/Makefile.local | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Makefile.local b/lib/Makefile.local index c56cba9..c83f387 100644 --- a/lib/Makefile.local +++ b/lib/Makefile.local @@ -27,7 +27,7 @@ LIBRARY_SUFFIX = dylib LINKER_NAME = libnotmuch.$(LIBRARY_SUFFIX) SONAME = libnotmuch.$(LIBNOTMUCH_VERSION_MAJOR).$(LIBRARY_SUFFIX) LIBNAME = libnotmuch.$(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE).$(LIBRARY_SUFFIX) -LIBRARY_LINK_FLAG = -dynamiclib -install_name $(SONAME) -compatibility_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR) -current_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE) +LIBRARY_LINK_FLAG = -dynamiclib -install_name $(libdir)/$(SONAME) -compatibility_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR) -current_version $(LIBNOTMUCH_VERSION_MAJOR).$(LIBNOTMUCH_VERSION_MINOR).$(LIBNOTMUCH_VERSION_RELEASE) else LIBRARY_SUFFIX = so LINKER_NAME = libnotmuch.$(LIBRARY_SUFFIX) -- 1.8.5.2 (Apple Git-48) ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
