Hello community, here is the log from the commit of package offlineimap for openSUSE:Factory checked in at 2017-11-13 14:10:24 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/offlineimap (Old) and /work/SRC/openSUSE:Factory/.offlineimap.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "offlineimap" Mon Nov 13 14:10:24 2017 rev:42 rq:541234 version:7.1.4 Changes: -------- --- /work/SRC/openSUSE:Factory/offlineimap/offlineimap.changes 2017-07-25 11:41:38.945729569 +0200 +++ /work/SRC/openSUSE:Factory/.offlineimap.new/offlineimap.changes 2017-11-13 14:10:44.055788169 +0100 @@ -1,0 +2,16 @@ +Mon Nov 13 08:41:28 UTC 2017 - [email protected] + +- update to 7.1.4 +* utf8: implement utf8foldernames option +* utf8: document new feature, deprecate old one +* remotehost should not be required if transporttunnel is used +* accounts: error out when no folder to sync +* sqlite: provide better message error for insert. +* folder: Gmail: fix copyright header. +* Remove some unnecessary whitespace +* folder: Gmail: remove dead code +* utf8foldernames: fix missing decode argument. +* Fix: if any tunnel (preauth_tunnel or transport_tunnel) + the hostname should not be requird + +------------------------------------------------------------------- Old: ---- offlineimap-7.1.2.tar.gz New: ---- offlineimap-7.1.4.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ offlineimap.spec ++++++ --- /var/tmp/diff_new_pack.UxJEcg/_old 2017-11-13 14:10:46.359705254 +0100 +++ /var/tmp/diff_new_pack.UxJEcg/_new 2017-11-13 14:10:46.359705254 +0100 @@ -19,7 +19,7 @@ %{!?_userunitdir:%define _userunitdir /usr/lib/systemd/user} Name: offlineimap -Version: 7.1.2 +Version: 7.1.4 Release: 0 Summary: IMAP/Maildir Synchronization Tool License: GPL-2.0+ ++++++ offlineimap-7.1.2.tar.gz -> offlineimap-7.1.4.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/Changelog.md new/offlineimap-7.1.4/Changelog.md --- old/offlineimap-7.1.2/Changelog.md 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/Changelog.md 2017-10-29 15:23:29.000000000 +0100 @@ -15,6 +15,98 @@ * The following excerpt is only usefull when rendered in the website. {:toc} +### OfflineIMAP v7.1.4 (2017-10-29) + +#### Notes + +Here is a bugfix release for v7.1.3. Two regressions got fixes and the +--delete-folder CLI option now expects an UTF-8 folder name when utf8foldernames +is enabled. + +This release was tested by: + +- Nicolas Sebrecht + +#### Authors + +- Nicolas Sebrecht (5) +- Thomas Merkel (1) + +#### Fixes + +- utf8foldernames: fix missing decode argument. [Nicolas Sebrecht] +- Fix: if any tunnel (preauth_tunnel or transport_tunnel) the hostname should not be required. [Thomas Merkel] + +#### Changes + +- utf8foldernames: support --delete-folder with UTF-8 folder name. [Nicolas Sebrecht] +- contrib/release.py improvements + + +### OfflineIMAP v7.1.3 (2017-10-08) + +#### Notes + +This release introduces a new experimental utf8foldernames configuration option. + +We already had the "tricky" decodefoldernames which is now deprecated. The new +code is the correct implementation for this feature. The changes are neat and +rather small. All the users having decodefoldernames are requested to move to +utf8foldernames. This requires to update almost all the functions like +nametrans, folderfilter, etc, because they work on the UTF-8 encoding. See the +documentation for more. Thank you Urs Liska for this contribution! + +In the long run, the idea is to: + +1. Remove decodefoldernames in favour of utf8foldernames. +2. Promote utf8foldernames up to stable. +3. Turn utf8foldernames on by default. + +Currently, folders with non-ASCII characters in their name have to be fully +re-downloaded. So, there's a bit more work to be done to have (3) and maybe (2). + +Also, this release includes a fix about remotehost and transporttunnel that +would require some testing. Thanks Thomas Merkel! + +There are documentation improvements, improved errors and minor code cleanups, +too. + +This release was tested by: + +- Nicolas Sebrecht +- Remi Locherer + + +#### Authors + +- Nicolas Sebrecht (11) +- Urs Liska (8) +- Thomas Merkel (1) + +#### Features + +- utf8: implement utf8foldernames option. [Urs Liska] +- utf8: document new feature, deprecate old one. [Urs Liska] + +#### Fixes + +- remotehost should not be required if transporttunnel is used. [Thomas Merkel] +- accounts: error out when no folder to sync. [Nicolas Sebrecht] +- sqlite: provide better message error for insert. [Nicolas Sebrecht] +- folder: Gmail: fix copyright header. [Nicolas Sebrecht] + +#### Changes + +- man: remove mention of experimental support for python 3. [Nicolas Sebrecht] +- man: mention the supported directions of the syncs. [Nicolas Sebrecht] +- folder: Gmail: remove dead code. [Nicolas Sebrecht] +- upcoming.py: get header template from external file. [Nicolas Sebrecht] +- upcoming.py: display a message with the filename once written. [Nicolas Sebrecht] +- contrib/helpers: sort testers by name. [Nicolas Sebrecht] +- Remove some unnecessary whitespace (in existing code). [Urs Liska] +- MAINTAINERS: Rainer is not currently active. [Nicolas Sebrecht] + + ### OfflineIMAP v7.1.2 (2017-07-10) #### Notes diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/MAINTAINERS.rst new/offlineimap-7.1.4/MAINTAINERS.rst --- old/offlineimap-7.1.2/MAINTAINERS.rst 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/MAINTAINERS.rst 2017-10-29 15:23:29.000000000 +0100 @@ -51,11 +51,6 @@ - github: nicolas33 - system: Linux -- Rainer M Krug - - email: Rainer at krugs.de - - github: rkrug - - system: OSX - - Remi Locherer - email: remi.locherer at relo.ch - system: OpenBSD maintainer @@ -79,7 +74,6 @@ - "J" - Łukasz Żarnowiecki - Nicolas Sebrecht -- Rainer M Krug - Remi Locherer diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/contrib/helpers.py new/offlineimap-7.1.4/contrib/helpers.py --- old/offlineimap-7.1.2/contrib/helpers.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/contrib/helpers.py 2017-10-29 15:23:29.000000000 +0100 @@ -83,7 +83,7 @@ @staticmethod def commit(msg): - cmd = shlex.split("git commit -s -m 'v{}'".format(msg)) + cmd = shlex.split("git commit -s -m '{}'".format(msg)) return run(cmd).decode(ENCODING) @staticmethod @@ -230,7 +230,7 @@ answer = User.request(msg, prompt).lower() if answer in ['y', 'yes']: return True - if defaultToYes and answer not in ['n', 'no']: + if defaultToYes is not False and answer not in ['n', 'no']: return True return False @@ -278,6 +278,7 @@ email = tester['email'] feedback = tester['feedback'] self.testers.append(Tester(name, email, feedback)) + self.testers.sort(key=lambda x: x.getName().lower()) @staticmethod def listTestersInTeam(): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/contrib/release.py new/offlineimap-7.1.4/contrib/release.py --- old/offlineimap-7.1.2/contrib/release.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/contrib/release.py 2017-10-29 15:23:29.000000000 +0100 @@ -221,7 +221,7 @@ self.version = version def setHeaders(self, messageId, date): - self.fd.write("Message-Id: <{}>\n".format(messageId)) + self.fd.write("Message-Id: {}\n".format(messageId)) self.fd.write("Date: {}\n".format(date)) self.fd.write("From: Nicolas Sebrecht <[email protected]>\n") self.fd.write("To: {}\n".format(MAILING_LIST)) @@ -261,8 +261,8 @@ class Website(object): def updateAPI(self): - req = "update API of the website? (requires {}) [Y/n]".format(SPHINXBUILD) - if not User.yesNo(req, defaultToYes=True): + req = "update API of the website? (requires {})".format(SPHINXBUILD) + if User.yesNo(req, defaultToYes=True) is False: return False if check_call(shlex.split("{} --version".format(SPHINXBUILD))) != 0: @@ -309,7 +309,7 @@ return Git.checkout(branchName, create=True) - Git.add('_doc/versions') + Git.add('.') Git.commit("update for offlineimap v{}".format(version)) User.pause( @@ -394,7 +394,7 @@ def make(self): Git.add('offlineimap/__init__.py') Git.add('Changelog.md') - commitMsg = "{}\n".format(newVersion) + commitMsg = "v{}\n".format(newVersion) for tester in self.testers.getListOk(): commitMsg = "{}\nTested-by: {} {}".format( commitMsg, tester.getName(), tester.getEmail() @@ -451,9 +451,9 @@ websiteBranch = release.getWebsiteBranch() print(END_MESSAGE.format( - announce=ANNOUNCE_FILE, - new_version=newVersion, - website_branch=websiteBranch) + announce=ANNOUNCE_FILE, + new_version=newVersion, + website_branch=websiteBranch) ) except Exception as e: release.restore() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/contrib/upcoming.py new/offlineimap-7.1.4/contrib/upcoming.py --- old/offlineimap-7.1.2/contrib/upcoming.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/contrib/upcoming.py 2017-10-29 15:23:29.000000000 +0100 @@ -17,34 +17,36 @@ UPCOMING_FILE = "{}/upcoming.txt".format(CACHEDIR) +UPCOMING_HEADER = "{}/upcoming-header.txt".format(CACHEDIR) -UPCOMING_HEADER = """ -Message-Id: <{messageId}> -Date: {date} -From: {name} <{email}> -To: {mailinglist} -Cc: {ccList} -Subject: [ANNOUNCE] upcoming offlineimap v{expectedVersion} - -# Notes - -I think it's time for a new release. - -I aim to make the new release in one week, approximately. If you'd like more -time, please let me know. ,-) - -Please, send me a mail to confirm it works for you. This will be written in the -release notes and the git logs. - - -# Authors - -""" +# Header is like: +# +#Message-Id: <{messageId}> +#Date: {date} +#From: {name} <{email}> +#To: {mailinglist} +#Cc: {ccList} +#Subject: [ANNOUNCE] upcoming offlineimap v{expectedVersion} +# +## Notes +# +#I think it's time for a new release. +# +#I aim to make the new release in one week, approximately. If you'd like more +#time, please let me know. ,-) +# +#Please, send me a mail to confirm it works for you. This will be written in the +#release notes and the git logs. +# +# +## Authors +# if __name__ == '__main__': offlineimapInfo = OfflineimapInfo() + print("Will read headers from {}".format(UPCOMING_HEADER)) Git.chdirToRepositoryTopLevel() oVersion = offlineimapInfo.getVersion() ccList = Testers.listTestersInTeam() @@ -54,7 +56,8 @@ if email not in ccList: ccList.append(email) - with open(UPCOMING_FILE, 'w') as upcoming: + with open(UPCOMING_FILE, 'w') as upcoming, \ + open(UPCOMING_HEADER, 'r') as fd_header: header = {} header['messageId'] = Git.buildMessageId() @@ -64,7 +67,7 @@ header['expectedVersion'] = User.request("Expected new version?") header['ccList'] = ", ".join(ccList) - upcoming.write(UPCOMING_HEADER.format(**header).lstrip()) + upcoming.write(fd_header.read().format(**header).lstrip()) upcoming.write(Git.getShortlog(oVersion)) upcoming.write("\n\n# Diffstat\n\n") @@ -72,3 +75,4 @@ upcoming.write("\n\n\n-- \n{}\n".format(Git.getLocalUser()[0])) system("{} {}".format(EDITOR, UPCOMING_FILE)) + print("{} written".format(UPCOMING_FILE)) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/docs/offlineimap.txt new/offlineimap-7.1.4/docs/offlineimap.txt --- old/offlineimap-7.1.2/docs/offlineimap.txt 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/docs/offlineimap.txt 2017-10-29 15:23:29.000000000 +0100 @@ -4,7 +4,7 @@ NAME ---- -offlineimap - Synchronize mailboxes and Maildirs +offlineimap - Synchronize mailboxes and Maildirs both ways or one either way. SYNOPSIS -------- @@ -18,7 +18,7 @@ account has two sides. One of the side must be an IMAP server. The other side can either be a Maildir or another IMAP server. -Python 3 is supported while still EXPERIMENTAL. +Works with Python 2. OPTIONS @@ -160,9 +160,10 @@ --delete-folder:: Delete a folder on the remote repository. + -Only one account must be specified/configured for this feature to work. The -folder name must be provided in IMAP encoding with the remote separators (likely -'/'). E.g.: "Remote/folder/name". +Only one account must be specified/configured for this feature to work or you +must provide one account with -a. The folder name must be provided with the +remote separators (likely '/') in UTF-8 if utf8foldernames is enabled or in IMAP +otherwise. E.g.: "Remote/folder/name". --migrate-fmd5-using-nametrans:: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/__init__.py new/offlineimap-7.1.4/offlineimap/__init__.py --- old/offlineimap-7.1.2/offlineimap/__init__.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/__init__.py 2017-10-29 15:23:29.000000000 +0100 @@ -2,7 +2,7 @@ __productname__ = 'OfflineIMAP' # Expecting trailing "-rcN" or "" for stable releases. -__version__ = "7.1.2" +__version__ = "7.1.4" __copyright__ = "Copyright 2002-2017 John Goerzen & contributors" __author__ = "John Goerzen" __author_email__= "[email protected]" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/accounts.py new/offlineimap-7.1.4/offlineimap/accounts.py --- old/offlineimap-7.1.2/offlineimap/accounts.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/accounts.py 2017-10-29 15:23:29.000000000 +0100 @@ -69,6 +69,8 @@ self.name = name self.metadatadir = config.getmetadatadir() self.localeval = config.getlocaleval() + # Store utf-8 support as a property of Account object + self.utf_8_support = self.getconfboolean('utf8foldernames', False) # Current :mod:`offlineimap.ui`, can be used for logging: self.ui = getglobalui() self.refreshperiod = self.getconffloat('autorefresh', 0.0) @@ -324,6 +326,15 @@ hook = self.getconf('presynchook', '') self.callhook(hook) + if self.utf_8_support and self.remoterepos.getdecodefoldernames(): + raise OfflineImapError("Configuration mismatch in account " + + "'%s'. "% self.getname() + + "\nAccount setting 'utf8foldernames' and repository " + + "setting 'decodefoldernames'\nmay not be used at the " + + "same time. This account has not been synchronized.\n" + + "Please check the configuration and documentation.", + OfflineImapError.ERROR.REPO) + quickconfig = self.getconfint('quick', 0) if quickconfig < 0: quick = True @@ -338,6 +349,7 @@ quick = False try: + startedThread = False remoterepos = self.remoterepos localrepos = self.localrepos statusrepos = self.statusrepos @@ -360,7 +372,7 @@ if not remotefolder.sync_this: self.ui.debug('', "Not syncing filtered folder '%s'" - "[%s]"% (remotefolder, remoterepos)) + "[%s]"% (remotefolder.getname(), remoterepos)) continue # Ignore filtered folder. # The remote folder names must not have the local sep char in @@ -378,7 +390,7 @@ localfolder = self.get_local_folder(remotefolder) if not localfolder.sync_this: self.ui.debug('', "Not syncing filtered folder '%s'" - "[%s]"% (localfolder, localfolder.repository)) + "[%s]"% (localfolder.getname(), localfolder.repository)) continue # Ignore filtered folder. if not globals.options.singlethreading: @@ -394,10 +406,15 @@ folderthreads.append(thread) else: syncfolder(self, remotefolder, quick) + startedThread = True # Wait for all threads to finish. for thr in folderthreads: thr.join() - mbnames.writeIntermediateFile(self.name) # Write out mailbox names. + if startedThread is True: + mbnames.writeIntermediateFile(self.name) # Write out mailbox names. + else: + msg = "Account {}: no folder to sync (folderfilter issue?)".format(self) + raise OfflineImapError(msg, OfflineImapError.ERROR.REPO) localrepos.forgetfolders() remoterepos.forgetfolders() except: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/folder/Gmail.py new/offlineimap-7.1.4/offlineimap/folder/Gmail.py --- old/offlineimap-7.1.2/offlineimap/folder/Gmail.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/folder/Gmail.py 2017-10-29 15:23:29.000000000 +0100 @@ -1,5 +1,5 @@ # Gmail IMAP folder support -# Copyright (C) 2002-2016 John Goerzen & contributors. +# Copyright (C) 2002-2017 John Goerzen & contributors. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -15,6 +15,8 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +"""Folder implementation to support features of the Gmail IMAP server.""" + import re import six from sys import exc_info @@ -23,7 +25,6 @@ import offlineimap.accounts from .IMAP import IMAPFolder -"""Folder implementation to support features of the Gmail IMAP server.""" class GmailFolder(IMAPFolder): """Folder implementation to support features of the Gmail IMAP server. @@ -41,11 +42,8 @@ https://developers.google.com/google-apps/gmail/imap_extensions """ - def __init__(self, imapserver, name, repository): - super(GmailFolder, self).__init__(imapserver, name, repository) - self.trash_folder = repository.gettrashfolder(name) - # Gmail will really delete messages upon EXPUNGE in these folders - self.real_delete_folders = [self.trash_folder, repository.getspamfolder()] + def __init__(self, imapserver, name, repository, decode=True): + super(GmailFolder, self).__init__(imapserver, name, repository, decode) # The header under which labels are stored self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/folder/IMAP.py new/offlineimap-7.1.4/offlineimap/folder/IMAP.py --- old/offlineimap-7.1.2/offlineimap/folder/IMAP.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/folder/IMAP.py 2017-10-29 15:23:29.000000000 +0100 @@ -41,12 +41,21 @@ class IMAPFolder(BaseFolder): - def __init__(self, imapserver, name, repository): - # FIXME: decide if unquoted name is from the responsability of the - # caller or not, but not both. + def __init__(self, imapserver, name, repository, decode=True): + # decode the folder name from IMAP4_utf_7 to utf_8 if + # - utf8foldernames is enabled for the *account* + # - the decode argument is given + # (default True is used when the folder name is the result of + # querying the IMAP server, while False is used when creating + # a folder object from a locally available utf_8 name) + # In any case the given name is first dequoted. name = imaputil.dequote(name) + if decode and repository.account.utf_8_support: + name = imaputil.IMAP_utf8(name) self.sep = imapserver.delim super(IMAPFolder, self).__init__(name, repository) + if repository.getdecodefoldernames(): + self.visiblename = imaputil.decode_mailbox_name(self.visiblename) self.idle_mode = False self.expunge = repository.getexpunge() self.root = None # imapserver.root @@ -67,7 +76,6 @@ if self.repository.getidlefolders(): self.idle_mode = True - def __selectro(self, imapobj, force=False): """Select this folder when we do not need write access. @@ -78,9 +86,15 @@ :param: Enforce new SELECT even if we are on that folder already. :returns: raises :exc:`OfflineImapError` severity FOLDER on error""" try: - imapobj.select(self.getfullname(), force = force) + imapobj.select(self.getfullIMAPname(), force=force) except imapobj.readonly: - imapobj.select(self.getfullname(), readonly = True, force = force) + imapobj.select(self.getfullIMAPname(), readonly=True, force=force) + + def getfullIMAPname(self): + name = self.getfullname() + if self.repository.account.utf_8_support: + name = imaputil.utf8_IMAP(name) + return name # Interface from BaseFolder def suggeststhreads(self): @@ -145,7 +159,7 @@ imapobj = self.imapserver.acquireconnection() try: # Select folder and get number of messages. - restype, imapdata = imapobj.select(self.getfullname(), True, + restype, imapdata = imapobj.select(self.getfullIMAPname(), True, True) self.imapserver.releaseconnection(imapobj) except OfflineImapError as e: @@ -211,7 +225,7 @@ res_data.remove(0) return res_data - res_type, imapdata = imapobj.select(self.getfullname(), True, True) + res_type, imapdata = imapobj.select(self.getfullIMAPname(), True, True) if imapdata == [None] or imapdata[0] == '0': # Empty folder, no need to populate message list. return None @@ -291,13 +305,6 @@ self.ui.messagelistloaded(self.repository, self, self.getmessagecount()) # Interface from BaseFolder - def getvisiblename(self): - vname = super(IMAPFolder, self).getvisiblename() - if self.repository.getdecodefoldernames(): - return imaputil.decode_mailbox_name(vname) - return vname - - # Interface from BaseFolder def getmessage(self, uid): """Retrieve message with UID from the IMAP server (incl body). @@ -635,7 +642,7 @@ try: # Select folder for append and make the box READ-WRITE. - imapobj.select(self.getfullname()) + imapobj.select(self.getfullIMAPname()) except imapobj.readonly: # readonly exception. Return original uid to notify that # we did not save the message. (see savemessage in Base.py) @@ -644,7 +651,7 @@ # Do the APPEND. try: - (typ, dat) = imapobj.append(self.getfullname(), + (typ, dat) = imapobj.append(self.getfullIMAPname(), imaputil.flagsmaildir2imap(flags), date, content) # This should only catch 'NO' responses since append() # will raise an exception for 'BAD' responses: @@ -758,7 +765,7 @@ fails_left = retry_num # Retry on dropped connection. while fails_left: try: - imapobj.select(self.getfullname(), readonly=True) + imapobj.select(self.getfullIMAPname(), readonly=True) res_type, data = imapobj.uid('fetch', uids, query) break except imapobj.abort as e: @@ -818,7 +825,7 @@ - field: field name to be stored/updated - data: field contents """ - imapobj.select(self.getfullname()) + imapobj.select(self.getfullIMAPname()) res_type, retdata = imapobj.uid('store', uid, field, data) if res_type != 'OK': severity = OfflineImapError.ERROR.MESSAGE @@ -879,7 +886,7 @@ imapobj = self.imapserver.acquireconnection() try: try: - imapobj.select(self.getfullname()) + imapobj.select(self.getfullIMAPname()) except imapobj.readonly: self.ui.flagstoreadonly(self, uidlist, flags) return @@ -954,7 +961,7 @@ imapobj = self.imapserver.acquireconnection() try: try: - imapobj.select(self.getfullname()) + imapobj.select(self.getfullIMAPname()) except imapobj.readonly: self.ui.deletereadonly(self, uidlist) return diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/folder/LocalStatusSQLite.py new/offlineimap-7.1.4/offlineimap/folder/LocalStatusSQLite.py --- old/offlineimap-7.1.2/offlineimap/folder/LocalStatusSQLite.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/folder/LocalStatusSQLite.py 2017-10-29 15:23:29.000000000 +0100 @@ -352,8 +352,14 @@ self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime, 'mtime': mtime, 'labels': labels} flags = ''.join(sorted(flags)) labels = ', '.join(sorted(labels)) - self.__sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)', - (uid,flags,mtime,labels)) + try: + self.__sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)', + (uid,flags,mtime,labels)) + except Exception as e: + six.reraise(UserWarning, + UserWarning("%s while inserting UID %s"% + (str(e), str(uid))), + exc_info()[2]) return uid diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/folder/UIDMaps.py new/offlineimap-7.1.4/offlineimap/folder/UIDMaps.py --- old/offlineimap-7.1.2/offlineimap/folder/UIDMaps.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/folder/UIDMaps.py 2017-10-29 15:23:29.000000000 +0100 @@ -40,8 +40,8 @@ diskr2l: dict mapping message uids: self.r2l[remoteuid]=localuid diskl2r: dict mapping message uids: self.r2l[localuid]=remoteuid""" - def __init__(self, *args, **kwargs): - IMAPFolder.__init__(self, *args, **kwargs) + def __init__(self, imapserver, name, repository, decode=True): + IMAPFolder.__init__(self, imapserver, name, repository, decode=False) self.dryrun = self.config.getdefaultboolean("general", "dry-run", True) self.maplock = Lock() self.diskr2l, self.diskl2r = self._loadmaps() @@ -49,7 +49,7 @@ # Representing the local IMAP Folder using local UIDs. # XXX: This should be removed since we inherit from IMAPFolder. # See commit 3ce514e92ba7 to know more. - self._mb = IMAPFolder(*args, **kwargs) + self._mb = IMAPFolder(imapserver, name, repository, decode=False) def _getmapfilename(self): return os.path.join(self.repository.getmapdir(), diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/imapserver.py new/offlineimap-7.1.4/offlineimap/imapserver.py --- old/offlineimap-7.1.2/offlineimap/imapserver.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/imapserver.py 2017-10-29 15:23:29.000000000 +0100 @@ -92,7 +92,7 @@ self.af = socket.AF_INET else: self.af = socket.AF_UNSPEC - self.hostname = None if self.preauth_tunnel else repos.gethost() + self.hostname = None if self.transport_tunnel or self.preauth_tunnel else repos.gethost() self.port = repos.getport() if self.port is None: self.port = 993 if self.usessl else 143 @@ -797,7 +797,7 @@ localrepos = account.localrepos remoterepos = account.remoterepos statusrepos = account.statusrepos - remotefolder = remoterepos.getfolder(self.folder) + remotefolder = remoterepos.getfolder(self.folder, decode=False) hook = account.getconf('presynchook', '') account.callhook(hook) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/imaputil.py new/offlineimap-7.1.4/offlineimap/imaputil.py --- old/offlineimap-7.1.2/offlineimap/imaputil.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/imaputil.py 2017-10-29 15:23:29.000000000 +0100 @@ -17,6 +17,8 @@ import re import string +import binascii +import codecs from offlineimap.ui import getglobalui @@ -370,3 +372,85 @@ return ret.decode('utf-7').encode('utf-8') except (UnicodeDecodeError, UnicodeEncodeError): return name + +# Functionality to convert folder names encoded in IMAP_utf_7 to utf_8. +# This is achieved by defining 'imap4_utf_7' as a proper encoding scheme. + +# Public API, to be used in repository definitions + +def IMAP_utf8(foldername): + """Convert IMAP4_utf_7 encoded string to utf-8""" + return foldername.decode('imap4-utf-7').encode('utf-8') + +def utf8_IMAP(foldername): + """Convert utf-8 encoded string to IMAP4_utf_7""" + return foldername.decode('utf-8').encode('imap4-utf-7') + +# Codec definition + +def modified_base64(s): + s = s.encode('utf-16be') + return binascii.b2a_base64(s).rstrip('\n=').replace('/', ',') + +def doB64(_in, r): + if _in: + r.append('&%s-' % modified_base64(''.join(_in))) + del _in[:] + +def encoder(s): + r = [] + _in = [] + for c in s: + ordC = ord(c) + if 0x20 <= ordC <= 0x25 or 0x27 <= ordC <= 0x7e: + doB64(_in, r) + r.append(c) + elif c == '&': + doB64(_in, r) + r.append('&-') + else: + _in.append(c) + doB64(_in, r) + return (str(''.join(r)), len(s)) + +# decoding +def modified_unbase64(s): + b = binascii.a2b_base64(s.replace(',', '/') + '===') + return unicode(b, 'utf-16be') + +def decoder(s): + r = [] + decode = [] + for c in s: + if c == '&' and not decode: + decode.append('&') + elif c == '-' and decode: + if len(decode) == 1: + r.append('&') + else: + r.append(modified_unbase64(''.join(decode[1:]))) + decode = [] + elif decode: + decode.append(c) + else: + r.append(c) + + if decode: + r.append(modified_unbase64(''.join(decode[1:]))) + bin_str = ''.join(r) + return (bin_str, len(s)) + +class StreamReader(codecs.StreamReader): + def decode(self, s, errors='strict'): + return decoder(s) + +class StreamWriter(codecs.StreamWriter): + def decode(self, s, errors='strict'): + return encoder(s) + +def imap4_utf_7(name): + if name == 'imap4-utf-7': + return (encoder, decoder, StreamReader, StreamWriter) + + +codecs.register(imap4_utf_7) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/repository/Base.py new/offlineimap-7.1.4/offlineimap/repository/Base.py --- old/offlineimap-7.1.2/offlineimap/repository/Base.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/repository/Base.py 2017-10-29 15:23:29.000000000 +0100 @@ -1,6 +1,6 @@ """ Base repository support """ -# Copyright (C) 2002-2016 John Goerzen & contributors +# Copyright (C) 2002-2017 John Goerzen & contributors # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -165,7 +165,13 @@ def deletefolder(self, foldername): raise NotImplementedError - def getfolder(self, foldername): + def getfolder(self, foldername, decode=True): + """Get the folder for this repo. + + WARNING: the signature changes whether it's remote or local: + - remote types have the decode arg + - local types don't have the decode arg + """ raise NotImplementedError def sync_folder_structure(self, local_repo, status_repo): @@ -242,7 +248,7 @@ # Get IMAPFolder and see if the reverse nametrans works fine. # TODO: getfolder() works only because we succeed in getting # inexisting folders which I would like to change. Take care! - tmp_remotefolder = remote_repo.getfolder(remote_name) + tmp_remotefolder = remote_repo.getfolder(remote_name, decode=False) loop_name = tmp_remotefolder.getvisiblename().replace( remote_repo.getsep(), local_repo.getsep()) if local_name != loop_name: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/repository/Gmail.py new/offlineimap-7.1.4/offlineimap/repository/Gmail.py --- old/offlineimap-7.1.2/offlineimap/repository/Gmail.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/repository/Gmail.py 2017-10-29 15:23:29.000000000 +0100 @@ -19,6 +19,7 @@ from offlineimap.repository.IMAP import IMAPRepository from offlineimap import folder, OfflineImapError + class GmailRepository(IMAPRepository): """Gmail IMAP repository. @@ -87,9 +88,9 @@ def getpreauthtunnel(self): return None - def getfolder(self, foldername): + def getfolder(self, foldername, decode=True): return self.getfoldertype()(self.imapserver, foldername, - self) + self, decode) def getfoldertype(self): return folder.Gmail.GmailFolder diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap/repository/IMAP.py new/offlineimap-7.1.4/offlineimap/repository/IMAP.py --- old/offlineimap-7.1.2/offlineimap/repository/IMAP.py 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap/repository/IMAP.py 2017-10-29 15:23:29.000000000 +0100 @@ -1,6 +1,6 @@ """ IMAP repository support """ -# Copyright (C) 2002-2016 John Goerzen & contributors +# Copyright (C) 2002-2017 John Goerzen & contributors # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -428,10 +428,10 @@ # No strategy yielded a password! return None - def getfolder(self, foldername): + def getfolder(self, foldername, decode=True): """Return instance of OfflineIMAP representative folder.""" - return self.getfoldertype()(self.imapserver, foldername, self) + return self.getfoldertype()(self.imapserver, foldername, self, decode) def getfoldertype(self): return folder.IMAP.IMAPFolder @@ -480,8 +480,7 @@ flaglist = [x.lower() for x in imaputil.flagsplit(flags)] if '\\noselect' in flaglist: continue - foldername = imaputil.dequote(name) - retval.append(self.getfoldertype()(self.imapserver, foldername, + retval.append(self.getfoldertype()(self.imapserver, name, self)) # Add all folderincludes if len(self.folderincludes): @@ -489,7 +488,7 @@ try: for foldername in self.folderincludes: try: - imapobj.select(foldername, readonly=True) + imapobj.select(imaputil.utf8_IMAP(foldername), readonly=True) except OfflineImapError as e: # couldn't select this folderinclude, so ignore folder. if e.severity > OfflineImapError.ERROR.FOLDER: @@ -498,7 +497,7 @@ 'Invalid folderinclude:') continue retval.append(self.getfoldertype()( - self.imapserver, foldername, self)) + self.imapserver, foldername, self, decode=False)) finally: self.imapserver.releaseconnection(imapobj) @@ -525,6 +524,8 @@ def deletefolder(self, foldername): """Delete a folder on the IMAP server.""" + if self.account.utf_8_support: + foldername = imaputil.utf8_IMAP(foldername) imapobj = self.imapserver.acquireconnection() try: result = imapobj.delete(foldername) @@ -556,6 +557,9 @@ return imapobj = self.imapserver.acquireconnection() try: + if self.account.utf_8_support: + foldername = imaputil.utf8_IMAP(foldername) + result = imapobj.create(foldername) if result[0] != 'OK': raise OfflineImapError("Folder '%s'[%s] could not be created. " diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/offlineimap-7.1.2/offlineimap.conf new/offlineimap-7.1.4/offlineimap.conf --- old/offlineimap-7.1.2/offlineimap.conf 2017-07-10 16:58:50.000000000 +0200 +++ new/offlineimap-7.1.4/offlineimap.conf 2017-10-29 15:23:29.000000000 +0100 @@ -432,6 +432,55 @@ #proxy = SOCKS5:IP:9999 +# EXPERIMENTAL: This option stands in the [Account Test] section. +# +# IMAP defines an encoding for non-ASCII ("international") characters, and most +# IMAP servers store folder names in this encoding. Note that the IMAP 4rev1 +# specification (RFC 3501) allows both UTF-8 and modified UTF-7 folder names +# so it is *possible* that an IMAP server already uses UTF-8 encoded folder +# names. But usually Folders that are shown as, say, "Gäste" will be represented +# as "G&AOQ-ste", and by default will be synchronized like this by offlineIMAP. +# +# This option converts IMAP folder names from IMAP4-UTF-7 to UTF-8 and back +# in order to have nicely readable UTF-8 folder names in the local copy. +# +# WARNING: with this option enabled: +# - compatibility with any other version is NOT GUARANTED (including newer); +# - existing set-ups will probably break. +# - no support is provided. +# +# IMPORTANT: READ THIS SECTION if you intend to enable this feature for an +# EXISTING ACCOUNT that has already been synchronized! +# Enabling UTF-8 encoded folder names will change many things on the local +# repository of an account, so you really have to create a new local repository +# and review the configuration. The least that would happen otherwise is a +# duplication of all folders containing non-ASCII characters. +# But also the following functionality may change, so the configuration in the +# remote repository configuration has to be reviewed/updated: +# - decodefoldernames +# This option is replaced by utf8foldernames and must be removed +# If both utf8foldernames and decodefoldernames are enabled the synchronization +# for the given account is aborted before doing any changes. +# - nametrans +# With utf8foldernames enabled any nametrans function will operate on the +# UTF-8 encoded folder names, while even with decodefoldernames enabled they +# operate on the original IMAP4-UTF-7 encoded names. +# - folderfilter +# Folder filters still work on the untranslated names before applying a +# nametrans function, but still this operates on the UTF-8 encoded names. +# - folderincludes +# With utf8foldernames enabled this function expects UTF-8 encoded folder +# names. +# - foldersort +# With utf8foldernames enabled the folder names passed to the sorting routine +# will be the UTF encoded names. +# - idlefolders +# With utf8foldernames enabled folders passed to this function are expected to +# be UTF-8 encoded. +# +#utf8foldernames = no + + # TESTING: This option stands in the [Account Test] section. # # Use authproxy connection for this account. Useful to bypass the GFW in China. @@ -1005,7 +1054,7 @@ #reference = Mail -# This option stands in the [Repository RemoteExample] section. +# DEPRECATED: This option stands in the [Repository RemoteExample] section. # # IMAP defines an encoding for non-ASCII ("international") characters. Enable # this option if you want to decode them to the nowadays ubiquitous UTF-8. @@ -1013,10 +1062,26 @@ # Note that the IMAP 4rev1 specification (RFC 3501) allows both UTF-8 and # modified UTF-7 folder names. # +# This option converts IMAP folder names from IMAP4-UTF-7 to UTF-8. +# +# NOTE/LIMITATION: +# - The reencoding is applied *after* a nametrans function that may be given, +# so it is important to note that nametrans will work on the undecoded +# UTF-7 names. +# - This option only works from a remote IMAP to a local Maildir repository +# - It only works *once*, so it can only be used for one-off backups +# (see https://github.com/OfflineIMAP/offlineimap/issues/299 and especially +# https://github.com/OfflineIMAP/offlineimap/issues/299#issuecomment-331243827) +# # WARNING: with this option enabled: # - compatibility with any other version is NOT GUARANTED (including newer); # - no support is provided. # +# DEPRECATION: +# This option is only there for backward compatibility with existing set-ups. +# For newly created accounts please use the utf8foldernames option on account +# level. +# # This feature was merged because it's small changes in the code. However, this # might seriously decrease the stability of the program. That's why it will # likely never be marked stable. The approach is: if it works for you, you're
