Kenneth Loafman has proposed merging lp:~aaron-whitehouse/duplicity/bug_1620085_exclude-if-present-locked-folder into lp:duplicity.
Requested reviews: duplicity-team (duplicity-team) Related bugs: Bug #1620085 in Duplicity: "--exclude-if-present gives OSError looking for tag in locked folders" https://bugs.launchpad.net/duplicity/+bug/1620085 For more details, see: https://code.launchpad.net/~aaron-whitehouse/duplicity/bug_1620085_exclude-if-present-locked-folder/+merge/304924 -- Your team duplicity-team is requested to review the proposed merge of lp:~aaron-whitehouse/duplicity/bug_1620085_exclude-if-present-locked-folder into lp:duplicity.
=== modified file 'CHANGELOG' --- CHANGELOG 2016-08-22 11:11:55 +0000 +++ CHANGELOG 2016-09-05 14:30:45 +0000 @@ -1,19 +1,21 @@ -New in v0.8.00 (2016/07/??) ---------------------------- -* Merged in lp:~aaron-whitehouse/duplicity/remove-python26 - - Remove Python 2.6 support references and tests. -* Fixed bug #1600692 with patch from Wolfgang Rohdewald - - Allow symlink to have optional trailing slash during verify. -* Merged in lp:~aaron-whitehouse/duplicity/07-fix_deja_dup_error_on_locked_files - - Revert log.Error to log.Warn, as it was prior to the merge in rev 1224, - as this was affecting other applications (e.g. deja dup; Bug #1605939). -* Merged in lp:~mwilck/duplicity/duplicity +New in v0.7.11 (2016/09/??) +--------------------------- +* Fixed bugs #815510 and #1615480 + - Changed default --volsize to 200MB +* Merged in lp:~mstoll-de/duplicity/duplicity + - Backblaze announced a new domain for the b2 api + +New in v0.7.10 (2016/08/20) +--------------------------- +* Merged in lp:~mwilck/duplicity/0.7-series - Speedup of path_matches_glob() by about 8x. See - https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268 + https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332 for more details. +* Remove -w from setsid in functional tests. * Fixed conflict in merge from Martin Wilck and applied - https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301492 - merge fixes setsid usage in functional testing. +<<<<<<< TREE * Fixed bug #1612472 with patch from David Cuthbert - Restore from S3 fails with --with-prefix-archive if prefix includes '/' * Merged in lp:~arashad.ahamad/duplicity/duplicity_latest @@ -24,6 +26,24 @@ - Changed default --volsize to 200MB * Merged in lp:~mstoll-de/duplicity/duplicity - Backblaze announced a new domain for the b2 api +======= +* Fixed bug #1612472 with patch from David Cuthbert + - Restore from S3 fails with --with-prefix-archive if prefix includes '/' +* Merged in lp:~arashad.ahamad/duplicity/duplicity_latest + - Changes for connecting to IBM Bluemix ObjectStorage. See man page. +* Merged in lp:~fenisilius/duplicity/acd_init_mkdir + - Allow duplicity to create remote folder + + +New in v0.7.09 (2016/07/24) +--------------------------- +* Fixed bug #1600692 with patch from Wolfgang Rohdewald + - Allow symlink to have optional trailing slash during verify. +* Merged in lp:~aaron-whitehouse/duplicity/07-fix_deja_dup_error_on_locked_files + - Revert log.Error to log.Warn, as it was prior to the merge in rev 1224, + as this was affecting other applications (e.g. deja dup; Bug #1605939). +* Merged in lp:~duplicity-team/duplicity/po-updates +>>>>>>> MERGE-SOURCE New in v0.7.08 (2016/07/02) @@ -81,6 +101,9 @@ * Fixed bug #1570293 duplicity is very slow due to excessive fsync - removed flush() after write. - revert to previous version +* Merged in lp:~aaron-whitehouse/duplicity/07-fix_deja_dup_error_on_locked_files + - Revert log.Error to log.Warn, as it was prior to the merge in rev 1224, + as this was affecting other applications (e.g. deja dup; Bug #1605939). New in v0.7.07 (2016/04/10) @@ -333,9 +356,6 @@ * Merged in lp:~ed.so/duplicity/gpg.binary - new parameter --gpg-binary allows user to point to a different gpg binary, not necessarily in path -* Merged in lp:~ed.so/duplicity/gpg.binary - - new parameter --gpg-binary allows user to point to a different gpg binary, - not necessarily in path * Merged in lp:~aaron-whitehouse/duplicity/fix_POTFILES.in_and_run-tests - Fixed two filename references in po/POTFILES.in, a mistake which crept in in rev 1093 and caused testing/run-tests to fail with "IndexError: list index === modified file 'Changelog.GNU' --- Changelog.GNU 2016-08-22 11:11:55 +0000 +++ Changelog.GNU 2016-09-05 14:30:45 +0000 @@ -1,22 +1,48 @@ -2016-08-22 Kenneth Loafman <[email protected]> - - * Fixed bugs #815510 and #1615480 - - Changed default --volsize to 200MB - * Merged in lp:~mstoll-de/duplicity/duplicity - - Backblaze announced a new domain for the b2 api - -2016-08-18 Kenneth Loafman <[email protected]> - - * Merged in lp:~fenisilius/duplicity/acd_init_mkdir - - Allow duplicity to create remote folder - -2016-08-12 Kenneth Loafman <[email protected]> - - * Fixed bug #1612472 with patch from David Cuthbert - - Restore from S3 fails with --with-prefix-archive if prefix includes '/' - * Merged in lp:~arashad.ahamad/duplicity/duplicity_latest - - Changes for connecting to IBM Bluemix ObjectStorage. See man page. - +<<<<<<< TREE +2016-08-22 Kenneth Loafman <[email protected]> + + * Fixed bugs #815510 and #1615480 + - Changed default --volsize to 200MB + * Merged in lp:~mstoll-de/duplicity/duplicity + - Backblaze announced a new domain for the b2 api + +2016-08-18 Kenneth Loafman <[email protected]> + + * Merged in lp:~fenisilius/duplicity/acd_init_mkdir + - Allow duplicity to create remote folder + +2016-08-12 Kenneth Loafman <[email protected]> + + * Fixed bug #1612472 with patch from David Cuthbert + - Restore from S3 fails with --with-prefix-archive if prefix includes '/' + * Merged in lp:~arashad.ahamad/duplicity/duplicity_latest + - Changes for connecting to IBM Bluemix ObjectStorage. See man page. + +======= +2016-08-22 Kenneth Loafman <[email protected]> + + * Fixed bugs #815510 and #1615480 + - Changed default --volsize to 200MB + * Merged in lp:~mstoll-de/duplicity/duplicity + - Backblaze announced a new domain for the b2 api + +2016-08-20 Kenneth Loafman <[email protected]> + + * Prep for 0.7.10 + +2016-08-18 Kenneth Loafman <[email protected]> + + * Merged in lp:~fenisilius/duplicity/acd_init_mkdir + - Allow duplicity to create remote folder + +2016-08-12 Kenneth Loafman <[email protected]> + + * Fixed bug #1612472 with patch from David Cuthbert + - Restore from S3 fails with --with-prefix-archive if prefix includes '/' + * Merged in lp:~arashad.ahamad/duplicity/duplicity_latest + - Changes for connecting to IBM Bluemix ObjectStorage. See man page. + +>>>>>>> MERGE-SOURCE 2016-07-31 Kenneth Loafman <[email protected]> * Fixed conflict in merge from Martin Wilck and applied @@ -25,27 +51,24 @@ 2016-07-28 Kenneth Loafman <[email protected]> - * Merged in lp:~mwilck/duplicity/duplicity + * Merged in lp:~mwilck/duplicity/0.7-series - Speedup of path_matches_glob() by about 8x. See - https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268 + https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332 for more details. + * Remove -w from setsid in functional tests. 2016-07-24 Kenneth Loafman <[email protected]> * Merged in lp:~aaron-whitehouse/duplicity/07-fix_deja_dup_error_on_locked_files - Revert log.Error to log.Warn, as it was prior to the merge in rev 1224, as this was affecting other applications (e.g. deja dup; Bug #1605939). + * Prep for 0.7.09 2016-07-20 Kenneth Loafman <[email protected]> * Fixed bug #1600692 with patch from Wolfgang Rohdewald - Allow symlink to have optional trailing slash during verify. -2016-07-03 Kenneth Loafman <[email protected]> - - * Merged in lp:~aaron-whitehouse/duplicity/remove-python26 - - Remove Python 2.6 support references and tests. - 2016-07-02 Kenneth Loafman <[email protected]> * Merged in lp:~aaron-whitehouse/duplicity/PEP8_W503_fixes === modified file 'README' --- README 2016-07-02 17:12:58 +0000 +++ README 2016-09-05 14:30:45 +0000 @@ -19,7 +19,7 @@ REQUIREMENTS: - * Python v2.7 or later + * Python v2.6 or later * librsync v0.9.6 or later * GnuPG v1.x for encryption * python-lockfile v0.9 or later for concurrency locking === modified file 'README-REPO' --- README-REPO 2016-07-02 17:12:58 +0000 +++ README-REPO 2016-09-05 14:30:45 +0000 @@ -47,18 +47,35 @@ Please run all tests on your branch (run-tests) before proposing a merge, to ensure that all tests pass. The decorator @unittest.expectedFailure can be used to commit a known-failing test case without breaking the test suite, for -example to exhibit the behaviour in a bug report before it has been fixed. +example to exhibit the behaviour in a bug report before it has been fixed: + +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest + + +class TestClass(unittest.TestCase): + """Test class to show expectedFailure""" + + @unittest.expectedFailure + def test_expected_failure(self): + """Test behaviour of expectedFailure""" + self.assertEqual(1, 2) ----------------------------------------- Testing against multiple Python versions: ----------------------------------------- -Duplicity uses tox to make it easy to test your code against multiple -environments. Running tests using the commands above will automatically test -code against different supported environments, including the versions of -dependencies used by the Launchpad build system. - +Duplicity currently supports Python versions v2.6 or later. Duplicity uses tox +to make it easy to test your code against multiple Python versions. Running +tests using the commands above will automatically test code against both +Python v2.6 and v2.7, if you have both installed on your system. It will also +test against the versions of dependencies used by the Launchpad build system. You can test against a single environment, e.g. -tox -e py27 +tox -e py26 for example if you are working on fixing a bug, but please do a full run-tests before submitting a merge request. + +For instructions on installing Python v2.6 on newer versions of Ubuntu, see +https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes === modified file 'duplicity/selection.py' --- duplicity/selection.py 2016-07-27 11:56:07 +0000 +++ duplicity/selection.py 2016-09-05 14:30:45 +0000 @@ -447,10 +447,24 @@ def exclude_sel_func(path): # do not follow symbolic links when checking for file existence! - if path.isdir() and path.append(filename).exists(): - return 0 - else: - return None + if path.isdir(): + # First check path is read accessible + if not (os.access(path.name, os.R_OK)): + # Path is not read accessible + # ToDo: Ideally this error would only show if the folder + # was ultimately included by the full set of selection + # functions. Currently this will give an error for any + # locked directory within the folder being backed up. + log.Warn(_( + "Error accessing possibly locked file %s") % util.ufn( + path.name), + log.WarningCode.cannot_read, util.escape(path.name)) + if diffdir.stats: + diffdir.stats.Errors += 1 + elif path.append(filename).exists(): + return 0 + else: + return None if include == 0: sel_func = exclude_sel_func === modified file 'po/duplicity.pot' --- po/duplicity.pot 2016-08-18 16:06:51 +0000 +++ po/duplicity.pot 2016-09-05 14:30:45 +0000 @@ -8,7 +8,11 @@ msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: Kenneth Loafman <[email protected]>\n" +<<<<<<< TREE "POT-Creation-Date: 2016-08-18 10:50-0500\n" +======= +"POT-Creation-Date: 2016-08-20 14:02-0500\n" +>>>>>>> MERGE-SOURCE "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n" "Language-Team: LANGUAGE <[email protected]>\n" === modified file 'testing/__init__.py' --- testing/__init__.py 2016-07-02 19:39:05 +0000 +++ testing/__init__.py 2016-09-05 14:30:45 +0000 @@ -21,7 +21,11 @@ import os import sys import time -import unittest + +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest from duplicity import backend from duplicity import globals === modified file 'testing/functional/test_selection.py' --- testing/functional/test_selection.py 2015-07-31 08:22:31 +0000 +++ testing/functional/test_selection.py 2016-09-05 14:30:45 +0000 @@ -837,6 +837,57 @@ self.restore_and_check() +class TestTrailingSlash2(IncludeExcludeFunctionalTest): + """ This tests the behaviour of globbing strings with a trailing slash""" + # See Bug #1479545 (https://bugs.launchpad.net/duplicity/+bug/1479545) + + def test_no_trailing_slash(self): + """ Test that including 1.py works as expected""" + self.backup("full", "testfiles/select2", + options=["--include", "testfiles/select2/1.py", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1.py']]) + + def test_trailing_slash(self): + """ Test that globs with a trailing slash only match directories""" + # Regression test for Bug #1479545 + # (https://bugs.launchpad.net/duplicity/+bug/1479545) + self.backup("full", "testfiles/select2", + options=["--include", "testfiles/select2/1.py/", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, []) + + def test_include_files_not_subdirectories(self): + """ Test that a trailing slash glob followed by a * glob only matches + files and not subdirectories""" + self.backup("full", "testfiles/select2", + options=["--exclude", "testfiles/select2/*/", + "--include", "testfiles/select2/*", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1.doc', '1.py']]) + + def test_include_subdirectories_not_files(self): + """ Test that a trailing slash glob only matches directories""" + self.backup("full", "testfiles/select2", + options=["--include", "testfiles/select2/1/1sub1/**/", + "--exclude", "testfiles/select2/1/1sub1/**", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1'], ['1sub1'], + ['1sub1sub1', '1sub1sub2', '1sub1sub3']]) + + class TestGlobbingReplacement(IncludeExcludeFunctionalTest): """ This tests the behaviour of the extended shell globbing pattern replacement functions.""" # See the manual for a description of behaviours, but in summary: @@ -875,55 +926,85 @@ self.assertEqual(restored, self.expected_restored_tree) -class TestTrailingSlash(IncludeExcludeFunctionalTest): - """ This tests the behaviour of globbing strings with a trailing slash""" - # See Bug #1479545 (https://bugs.launchpad.net/duplicity/+bug/1479545) - - def test_no_trailing_slash(self): - """ Test that including 1.py works as expected""" - self.backup("full", "testfiles/select2", - options=["--include", "testfiles/select2/1.py", - "--exclude", "**"]) - self.restore() - restore_dir = 'testfiles/restore_out' - restored = self.directory_tree_to_list_of_lists(restore_dir) - self.assertEqual(restored, [['1.py']]) - - def test_trailing_slash(self): - """ Test that globs with a trailing slash only match directories""" - # ToDo: Bug #1479545 - # (https://bugs.launchpad.net/duplicity/+bug/1479545) - self.backup("full", "testfiles/select2", - options=["--include", "testfiles/select2/1.py/", - "--exclude", "**"]) - self.restore() - restore_dir = 'testfiles/restore_out' - restored = self.directory_tree_to_list_of_lists(restore_dir) - self.assertEqual(restored, []) - - def test_include_files_not_subdirectories(self): - """ Test that a trailing slash glob followed by a * glob only matches - files and not subdirectories""" - self.backup("full", "testfiles/select2", - options=["--exclude", "testfiles/select2/*/", - "--include", "testfiles/select2/*", - "--exclude", "**"]) - self.restore() - restore_dir = 'testfiles/restore_out' - restored = self.directory_tree_to_list_of_lists(restore_dir) - self.assertEqual(restored, [['1.doc', '1.py']]) - - def test_include_subdirectories_not_files(self): - """ Test that a trailing slash glob only matches directories""" - self.backup("full", "testfiles/select2", - options=["--include", "testfiles/select2/1/1sub1/**/", - "--exclude", "testfiles/select2/1/1sub1/**", - "--exclude", "**"]) - self.restore() - restore_dir = 'testfiles/restore_out' - restored = self.directory_tree_to_list_of_lists(restore_dir) - self.assertEqual(restored, [['1'], ['1sub1'], - ['1sub1sub1', '1sub1sub2', '1sub1sub3']]) +class TestExcludeIfPresent(IncludeExcludeFunctionalTest): + """ This tests the behaviour of duplicity's --exclude-if-present option""" + + def test_exclude_if_present_baseline(self): + """ Test that duplicity normally backs up files""" + with open("testfiles/select2/1/1sub1/1sub1sub1/.nobackup", "w") as tag: + tag.write("Files in this folder should not be backed up.") + self.backup("full", "testfiles/select2/1/1sub1", + options=["--include", "testfiles/select2/1/1sub1/1sub1sub1/*", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1sub1sub1'], + ['.nobackup', '1sub1sub1_file.txt']]) + + def test_exclude_if_present_excludes(self): + """ Test that duplicity excludes files with relevant tag""" + with open("testfiles/select2/1/1sub1/1sub1sub1/.nobackup", "w") as tag: + tag.write("Files in this folder should not be backed up.") + self.backup("full", "testfiles/select2/1/1sub1", + options=["--exclude-if-present", ".nobackup", + "--include", "testfiles/select2/1/1sub1/1sub1sub1/*", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, []) + + def test_exclude_if_present_excludes_2(self): + """ Test that duplicity excludes files with relevant tag""" + with open("testfiles/select2/1/1sub1/1sub1sub1/EXCLUDE.tag", "w") as tag: + tag.write("Files in this folder should also not be backed up.") + self.backup("full", "testfiles/select2/1/1sub1", + options=["--exclude-if-present", "EXCLUDE.tag", + "--include", "testfiles/select2/1/1sub1/1sub1sub1/*", + "--exclude", "**"]) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, []) + + +class TestLockedFoldersNoError(IncludeExcludeFunctionalTest): + """ This tests that inaccessible folders do not cause an error""" + + def test_locked_baseline(self): + """ Test no error if locked in path but excluded""" + folder_to_lock = "testfiles/select2/1/1sub1/1sub1sub3" + initial_mode = os.stat(folder_to_lock).st_mode + os.chmod(folder_to_lock, 0o0000) + self.backup("full", "testfiles/select2/1/1sub1", + options=["--include", "testfiles/select2/1/1sub1/1sub1sub1/*", + "--exclude", "**"]) + os.chmod(folder_to_lock, initial_mode) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1sub1sub1'], + ['1sub1sub1_file.txt']]) + + def test_locked_excl_if_present(self): + """ Test no error if excluded locked with --exclude-if-present""" + # Regression test for Bug #1620085 + # https://bugs.launchpad.net/duplicity/+bug/1620085 + folder_to_lock = "testfiles/select2/1/1sub1/1sub1sub3" + initial_mode = os.stat(folder_to_lock).st_mode + os.chmod(folder_to_lock, 0o0000) + self.backup("full", "testfiles/select2/1/1sub1", + options=["--exclude-if-present", "EXCLUDE.tag", + "--include", "testfiles/select2/1/1sub1/1sub1sub1/*", + "--exclude", "**"]) + os.chmod(folder_to_lock, initial_mode) + self.restore() + restore_dir = 'testfiles/restore_out' + restored = self.directory_tree_to_list_of_lists(restore_dir) + self.assertEqual(restored, [['1sub1sub1'], + ['1sub1sub1_file.txt']]) + if __name__ == "__main__": unittest.main() === added file 'testing/gnupg/gpg.conf' --- testing/gnupg/gpg.conf 1970-01-01 00:00:00 +0000 +++ testing/gnupg/gpg.conf 2016-09-05 14:30:45 +0000 @@ -0,0 +1,9 @@ +# gpg2 doesn't try all secrets by default, so add this option +# Otherwise the tests with hidden encryption key will fail +try-all-secrets + +# gpg2 2.1.13 has a bug that prevents the line above from working +# (https://bugs.gnupg.org/gnupg/issue1985) +# Uncomment the line below if you have gnupg2 2.1.13 +# (but that line will break gpg 1.x, so we can't use it by default) +#try-secret-key 96B629431B77DC52B1917B40839E6A2856538CCF === removed file 'testing/gnupg/gpg.conf' --- testing/gnupg/gpg.conf 2016-08-01 14:57:22 +0000 +++ testing/gnupg/gpg.conf 1970-01-01 00:00:00 +0000 @@ -1,9 +0,0 @@ -# gpg2 doesn't try all secrets by default, so add this option -# Otherwise the tests with hidden encryption key will fail -try-all-secrets - -# gpg2 2.1.13 has a bug that prevents the line above from working -# (https://bugs.gnupg.org/gnupg/issue1985) -# Uncomment the line below if you have gnupg2 2.1.13 -# (but that line will break gpg 1.x, so we can't use it by default) -#try-secret-key 96B629431B77DC52B1917B40839E6A2856538CCF === modified file 'testing/test_code.py' --- testing/test_code.py 2016-07-02 19:39:05 +0000 +++ testing/test_code.py 2016-09-05 14:30:45 +0000 @@ -22,7 +22,10 @@ import sys import subprocess -import unittest +if sys.version_info < (2, 7): + import unittest2 as unittest # @UnresolvedImport @UnusedImport +else: + import unittest # @Reimport from . import _top_dir, DuplicityTestCase # @IgnorePep8 === modified file 'tox.ini' --- tox.ini 2016-07-02 19:33:34 +0000 +++ tox.ini 2016-09-05 14:30:45 +0000 @@ -1,5 +1,14 @@ [tox] -envlist=py27,lpbuildd-precise +envlist=py26,py27,lpbuildd-precise + +[testenv:py26] +setenv= + RUN_CODE_TESTS=0 +deps= + mock==0.7.2 + pexpect==2.4 + ptyprocess==0.4 + unittest2 [testenv:py27] setenv= @@ -20,5 +29,5 @@ {envpython} {toxinidir}/setup.py test {posargs} [pep8] -ignore=E402,E731 +ignore=E402,E501,E731,W503 max-line-length=120
_______________________________________________ Mailing list: https://launchpad.net/~duplicity-team Post to : [email protected] Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
