On 08.02.2015 15:46, Rupert Levene wrote: > On 8 February 2015 at 14:32, <[email protected]> wrote: >> On 08.02.2015 15:05, Rupert Levene wrote: >>> Hi, >>> >>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB >>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was >>> created, and duplicity chokes on this. It pauses for a while (I guess >>> while it downloads the file) and then says >>> >>> Attempt 1 failed: BackendException: Failed to download file >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote >>> folder 'i7-bigstore': join() result is too long for a Python string >>> Backtrace of previous error: Traceback (innermost last): >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line >>> 311, in iterate >>> return fn(*args, **kwargs) >>> File >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", >>> line 144, in get >>> % (remote_filename, self.folder.title.text, str(e)), raise_errors) >>> File >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", >>> line 182, in __handle_error >>> raise BackendException(message) >>> BackendException: Failed to download file >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote >>> folder 'i7-bigstore': join() result is too long for a Python string >>> >>> It'll then try again, and give the same error, at which point I hit >>> control-C. >>> >>> Version 0.7.01 fails more quickly: >>> >>> $ apt-cache policy duplicity >>> duplicity: >>> Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 >>> Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 >>> Version table: >>> *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0 >>> 500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/ >>> precise/main i386 Packages >>> 100 /var/lib/dpkg/status >>> 0.6.18-0ubuntu3.5 0 >>> 500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main >>> i386 Packages >>> 0.6.18-0ubuntu3 0 >>> 500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386 Packages >>> $ duplicity -V >>> duplicity 0.7.01 >>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist >>> /home/rupert/.duply/i7-bigstore/exclude >>> gdocs://[email protected]/backups/duplicity/i7-bigstore >>> /mnt/bigstore >>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx >>> Using backup name: xxxxxxxxxxxxxxxxxxxxx >>> Import of duplicity.backends.azurebackend Succeeded >>> Import of duplicity.backends.botobackend Succeeded >>> Import of duplicity.backends.cfbackend Succeeded >>> Import of duplicity.backends.copycombackend Succeeded >>> Import of duplicity.backends.dpbxbackend Succeeded >>> Import of duplicity.backends.gdocsbackend Succeeded >>> Import of duplicity.backends.giobackend Succeeded >>> Import of duplicity.backends.hsibackend Succeeded >>> Import of duplicity.backends.hubicbackend Succeeded >>> Import of duplicity.backends.imapbackend Succeeded >>> Import of duplicity.backends.lftpbackend Succeeded >>> Import of duplicity.backends.localbackend Succeeded >>> Import of duplicity.backends.megabackend Succeeded >>> Import of duplicity.backends.ncftpbackend Succeeded >>> Import of duplicity.backends.onedrivebackend Failed: No module named >>> requests >>> Import of duplicity.backends.par2backend Succeeded >>> Import of duplicity.backends.pydrivebackend Succeeded >>> Import of duplicity.backends.rsyncbackend Succeeded >>> Import of duplicity.backends.ssh_paramiko_backend Succeeded >>> Import of duplicity.backends.ssh_pexpect_backend Succeeded >>> Import of duplicity.backends.swiftbackend Succeeded >>> Import of duplicity.backends.sxbackend Succeeded >>> Import of duplicity.backends.tahoebackend Succeeded >>> Import of duplicity.backends.webdavbackend Succeeded >>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir >>> Traceback (most recent call last): >>> File "/usr/bin/duplicity", line 1497, in <module> >>> with_tempdir(main) >>> File "/usr/bin/duplicity", line 1491, in with_tempdir >>> fn() >>> File "/usr/bin/duplicity", line 1324, in main >>> action = commandline.ProcessCommandLine(sys.argv[1:]) >>> File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", >>> line 1056, in ProcessCommandLine >>> backup, local_pathname = set_backend(args[0], args[1]) >>> File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", >>> line 949, in set_backend >>> globals.backend = backend.get_backend(bend) >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line >>> 221, in get_backend >>> obj = get_backend_object(url_string) >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line >>> 207, in get_backend_object >>> return factory(pu) >>> File >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", >>> line 61, in __init__ >>> entries = self._fetch_entries(parent_folder_id, 'folder', folder_name) >>> File >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", >>> line 161, in _fetch_entries >>> entries = self.client.get_all_resources(uri=uri) >>> AttributeError: 'DocsClient' object has no attribute 'get_all_resources' >>> >>> Are these two independent bugs? Is there a workaround or fix? >>> >> >> yes, it looks like two different bugs with two different error messages. >> >> first try latest stable duplicity 0.6.25.. there were changes in the gdocs >> backend since your version. >> >> if that fails - can you try downloading your whole backups to a local folder >> and try to restore from there? >> >> ..ede/duply.net >> >> _______________________________________________ >> Duplicity-talk mailing list >> [email protected] >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk > > I had tried 0.6.25 too, and it was giving the same get_all_resources > error as 0.7.01. However since then I used pip to upgrade gdata, and > 0.7.01 gives me very similar output to my original error from 0.6.18: > > $ duplicity verify --verbosity '9' --exclude-globbing-filelist > /home/rupert/.duply/i7-bigstore/exclude > gdocs://[email protected]/backups/duplicity/i7-bigstore > /mnt/bigstore > Using archive dir: > /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > Import of duplicity.backends.azurebackend Succeeded > Import of duplicity.backends.botobackend Succeeded > Import of duplicity.backends.cfbackend Succeeded > Import of duplicity.backends.copycombackend Succeeded > Import of duplicity.backends.dpbxbackend Succeeded > Import of duplicity.backends.gdocsbackend Succeeded > Import of duplicity.backends.giobackend Succeeded > Import of duplicity.backends.hsibackend Succeeded > Import of duplicity.backends.hubicbackend Succeeded > Import of duplicity.backends.imapbackend Succeeded > Import of duplicity.backends.lftpbackend Succeeded > Import of duplicity.backends.localbackend Succeeded > Import of duplicity.backends.megabackend Succeeded > Import of duplicity.backends.ncftpbackend Succeeded > Import of duplicity.backends.onedrivebackend Failed: No module named requests > Import of duplicity.backends.par2backend Succeeded > Import of duplicity.backends.pydrivebackend Succeeded > Import of duplicity.backends.rsyncbackend Succeeded > Import of duplicity.backends.ssh_paramiko_backend Succeeded > Import of duplicity.backends.ssh_pexpect_backend Succeeded > Import of duplicity.backends.swiftbackend Succeeded > Import of duplicity.backends.sxbackend Succeeded > Import of duplicity.backends.tahoebackend Succeeded > Import of duplicity.backends.webdavbackend Succeeded > Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude > Main action: verify > ================================================================================ > duplicity 0.7.01 ($reldate) > Args: /usr/bin/duplicity verify --verbosity 9 > --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude > gdocs://[email protected]/backups/duplicity/i7-bigstore > /mnt/bigstore > Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed > Dec 17 00:38:38 UTC 2014 i686 i686 > /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52) > [GCC 4.6.3] > ================================================================================ > Synchronizing remote metadata to local cache... > Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local cache. > Using temporary directory /tmp/duplicity-L9vwHt-tempdir > Registering (mktemp) temporary file > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1 > Backtrace of previous error: Traceback (innermost last): > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 361, in inner_retry > return fn(self, *args) > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in > get > self.backend._get(remote_filename, local_path) > File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > line 110, in _get > self.client.DownloadResource(entry, local_path.name) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 369, in download_resource > self._download_file(uri, file_path, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 468, in _download_file > f.write(self._get_content(uri, **kwargs)) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 451, in _get_content > return server_response.read() > File "/usr/lib/python2.7/httplib.py", line 541, in read > return self._read_chunked(amt) > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > return ''.join(value) > OverflowError: join() result is too long for a Python string > > Attempt 1 failed. OverflowError: join() result is too long for a Python string > Backtrace of previous error: Traceback (innermost last): > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 361, in inner_retry > return fn(self, *args) > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in > get > self.backend._get(remote_filename, local_path) > File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > line 110, in _get > self.client.DownloadResource(entry, local_path.name) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 369, in download_resource > self._download_file(uri, file_path, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 468, in _download_file > f.write(self._get_content(uri, **kwargs)) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 451, in _get_content > return server_response.read() > File "/usr/lib/python2.7/httplib.py", line 541, in read > return self._read_chunked(amt) > File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked > value.append(self._safe_read(chunk_left)) > File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read > chunk = self.fp.read(min(amt, MAXAMOUNT)) > File "/usr/lib/python2.7/socket.py", line 396, in read > buf.write(data) > MemoryError: out of memory > > Attempt 2 failed. MemoryError: out of memory > Backtrace of previous error: Traceback (innermost last): > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 361, in inner_retry > return fn(self, *args) > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in > get > self.backend._get(remote_filename, local_path) > File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > line 110, in _get > self.client.DownloadResource(entry, local_path.name) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 369, in download_resource > self._download_file(uri, file_path, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 468, in _download_file > f.write(self._get_content(uri, **kwargs)) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 451, in _get_content > return server_response.read() > File "/usr/lib/python2.7/httplib.py", line 541, in read > return self._read_chunked(amt) > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > return ''.join(value) > OverflowError: join() result is too long for a Python string > > Attempt 3 failed. OverflowError: join() result is too long for a Python string > Backtrace of previous error: Traceback (innermost last): > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 361, in inner_retry > return fn(self, *args) > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in > get > self.backend._get(remote_filename, local_path) > File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > line 110, in _get > self.client.DownloadResource(entry, local_path.name) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 369, in download_resource > self._download_file(uri, file_path, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 468, in _download_file > f.write(self._get_content(uri, **kwargs)) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 451, in _get_content > return server_response.read() > File "/usr/lib/python2.7/httplib.py", line 541, in read > return self._read_chunked(amt) > File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked > value.append(self._safe_read(chunk_left)) > File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read > chunk = self.fp.read(min(amt, MAXAMOUNT)) > File "/usr/lib/python2.7/socket.py", line 400, in read > buf.write(data) > MemoryError: out of memory > > Attempt 4 failed. MemoryError: out of memory > Backtrace of previous error: Traceback (innermost last): > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 361, in inner_retry > return fn(self, *args) > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in > get > self.backend._get(remote_filename, local_path) > File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > line 110, in _get > self.client.DownloadResource(entry, local_path.name) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 369, in download_resource > self._download_file(uri, file_path, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 468, in _download_file > f.write(self._get_content(uri, **kwargs)) > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > line 451, in _get_content > return server_response.read() > File "/usr/lib/python2.7/httplib.py", line 541, in read > return self._read_chunked(amt) > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > return ''.join(value) > OverflowError: join() result is too long for a Python string > > Giving up after 5 attempts. OverflowError: join() result is too long > for a Python string > Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c> > Removing still remembered temporary file > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1 > $ > > I am running 32-bit linux with 4gb RAM and 4gb swap. > > I'll try downloading the backup to local storage as you suggest. >
please do.. looks like httplib fails on 2GB files and there isn't much we can do about that. Mike, Ken: any idea? ..ede/duply.net _______________________________________________ Mailing list: https://launchpad.net/~duplicity-team Post to : [email protected] Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
