Were there any errors during the initial backup? This looks like a 32-bit Python limit.
...Ken On Sun, Feb 8, 2015 at 9:16 AM, <[email protected]> wrote: > On 08.02.2015 15:46, Rupert Levene wrote: > > On 8 February 2015 at 14:32, <[email protected]> wrote: > >> On 08.02.2015 15:05, Rupert Levene wrote: > >>> Hi, > >>> > >>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB > >>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was > >>> created, and duplicity chokes on this. It pauses for a while (I guess > >>> while it downloads the file) and then says > >>> > >>> Attempt 1 failed: BackendException: Failed to download file > >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote > >>> folder 'i7-bigstore': join() result is too long for a Python string > >>> Backtrace of previous error: Traceback (innermost last): > >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > >>> 311, in iterate > >>> return fn(*args, **kwargs) > >>> File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > >>> line 144, in get > >>> % (remote_filename, self.folder.title.text, str(e)), raise_errors) > >>> File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > >>> line 182, in __handle_error > >>> raise BackendException(message) > >>> BackendException: Failed to download file > >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote > >>> folder 'i7-bigstore': join() result is too long for a Python string > >>> > >>> It'll then try again, and give the same error, at which point I hit > control-C. > >>> > >>> Version 0.7.01 fails more quickly: > >>> > >>> $ apt-cache policy duplicity > >>> duplicity: > >>> Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 > >>> Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 > >>> Version table: > >>> *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0 > >>> 500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/ > >>> precise/main i386 Packages > >>> 100 /var/lib/dpkg/status > >>> 0.6.18-0ubuntu3.5 0 > >>> 500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main > >>> i386 Packages > >>> 0.6.18-0ubuntu3 0 > >>> 500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386 > Packages > >>> $ duplicity -V > >>> duplicity 0.7.01 > >>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist > >>> /home/rupert/.duply/i7-bigstore/exclude > >>> gdocs://[email protected]/backups/duplicity/i7-bigstore > >>> /mnt/bigstore > >>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx > >>> Using backup name: xxxxxxxxxxxxxxxxxxxxx > >>> Import of duplicity.backends.azurebackend Succeeded > >>> Import of duplicity.backends.botobackend Succeeded > >>> Import of duplicity.backends.cfbackend Succeeded > >>> Import of duplicity.backends.copycombackend Succeeded > >>> Import of duplicity.backends.dpbxbackend Succeeded > >>> Import of duplicity.backends.gdocsbackend Succeeded > >>> Import of duplicity.backends.giobackend Succeeded > >>> Import of duplicity.backends.hsibackend Succeeded > >>> Import of duplicity.backends.hubicbackend Succeeded > >>> Import of duplicity.backends.imapbackend Succeeded > >>> Import of duplicity.backends.lftpbackend Succeeded > >>> Import of duplicity.backends.localbackend Succeeded > >>> Import of duplicity.backends.megabackend Succeeded > >>> Import of duplicity.backends.ncftpbackend Succeeded > >>> Import of duplicity.backends.onedrivebackend Failed: No module named > requests > >>> Import of duplicity.backends.par2backend Succeeded > >>> Import of duplicity.backends.pydrivebackend Succeeded > >>> Import of duplicity.backends.rsyncbackend Succeeded > >>> Import of duplicity.backends.ssh_paramiko_backend Succeeded > >>> Import of duplicity.backends.ssh_pexpect_backend Succeeded > >>> Import of duplicity.backends.swiftbackend Succeeded > >>> Import of duplicity.backends.sxbackend Succeeded > >>> Import of duplicity.backends.tahoebackend Succeeded > >>> Import of duplicity.backends.webdavbackend Succeeded > >>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir > >>> Traceback (most recent call last): > >>> File "/usr/bin/duplicity", line 1497, in <module> > >>> with_tempdir(main) > >>> File "/usr/bin/duplicity", line 1491, in with_tempdir > >>> fn() > >>> File "/usr/bin/duplicity", line 1324, in main > >>> action = commandline.ProcessCommandLine(sys.argv[1:]) > >>> File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", > >>> line 1056, in ProcessCommandLine > >>> backup, local_pathname = set_backend(args[0], args[1]) > >>> File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", > >>> line 949, in set_backend > >>> globals.backend = backend.get_backend(bend) > >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > >>> 221, in get_backend > >>> obj = get_backend_object(url_string) > >>> File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > >>> 207, in get_backend_object > >>> return factory(pu) > >>> File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > >>> line 61, in __init__ > >>> entries = self._fetch_entries(parent_folder_id, 'folder', > folder_name) > >>> File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > >>> line 161, in _fetch_entries > >>> entries = self.client.get_all_resources(uri=uri) > >>> AttributeError: 'DocsClient' object has no attribute > 'get_all_resources' > >>> > >>> Are these two independent bugs? Is there a workaround or fix? > >>> > >> > >> yes, it looks like two different bugs with two different error messages. > >> > >> first try latest stable duplicity 0.6.25.. there were changes in the > gdocs backend since your version. > >> > >> if that fails - can you try downloading your whole backups to a local > folder and try to restore from there? > >> > >> ..ede/duply.net > >> > >> _______________________________________________ > >> Duplicity-talk mailing list > >> [email protected] > >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk > > > > I had tried 0.6.25 too, and it was giving the same get_all_resources > > error as 0.7.01. However since then I used pip to upgrade gdata, and > > 0.7.01 gives me very similar output to my original error from 0.6.18: > > > > $ duplicity verify --verbosity '9' --exclude-globbing-filelist > > /home/rupert/.duply/i7-bigstore/exclude > > gdocs://[email protected]/backups/duplicity/i7-bigstore > > /mnt/bigstore > > Using archive dir: > > /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > Import of duplicity.backends.azurebackend Succeeded > > Import of duplicity.backends.botobackend Succeeded > > Import of duplicity.backends.cfbackend Succeeded > > Import of duplicity.backends.copycombackend Succeeded > > Import of duplicity.backends.dpbxbackend Succeeded > > Import of duplicity.backends.gdocsbackend Succeeded > > Import of duplicity.backends.giobackend Succeeded > > Import of duplicity.backends.hsibackend Succeeded > > Import of duplicity.backends.hubicbackend Succeeded > > Import of duplicity.backends.imapbackend Succeeded > > Import of duplicity.backends.lftpbackend Succeeded > > Import of duplicity.backends.localbackend Succeeded > > Import of duplicity.backends.megabackend Succeeded > > Import of duplicity.backends.ncftpbackend Succeeded > > Import of duplicity.backends.onedrivebackend Failed: No module named > requests > > Import of duplicity.backends.par2backend Succeeded > > Import of duplicity.backends.pydrivebackend Succeeded > > Import of duplicity.backends.rsyncbackend Succeeded > > Import of duplicity.backends.ssh_paramiko_backend Succeeded > > Import of duplicity.backends.ssh_pexpect_backend Succeeded > > Import of duplicity.backends.swiftbackend Succeeded > > Import of duplicity.backends.sxbackend Succeeded > > Import of duplicity.backends.tahoebackend Succeeded > > Import of duplicity.backends.webdavbackend Succeeded > > Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude > > Main action: verify > > > ================================================================================ > > duplicity 0.7.01 ($reldate) > > Args: /usr/bin/duplicity verify --verbosity 9 > > --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude > > gdocs://[email protected]/backups/duplicity/i7-bigstore > > /mnt/bigstore > > Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed > > Dec 17 00:38:38 UTC 2014 i686 i686 > > /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52) > > [GCC 4.6.3] > > > ================================================================================ > > Synchronizing remote metadata to local cache... > > Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local > cache. > > Using temporary directory /tmp/duplicity-L9vwHt-tempdir > > Registering (mktemp) temporary file > > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1 > > Backtrace of previous error: Traceback (innermost last): > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > > 361, in inner_retry > > return fn(self, *args) > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 534, in get > > self.backend._get(remote_filename, local_path) > > File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > > line 110, in _get > > self.client.DownloadResource(entry, local_path.name) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 369, in download_resource > > self._download_file(uri, file_path, **kwargs) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 468, in _download_file > > f.write(self._get_content(uri, **kwargs)) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 451, in _get_content > > return server_response.read() > > File "/usr/lib/python2.7/httplib.py", line 541, in read > > return self._read_chunked(amt) > > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > > return ''.join(value) > > OverflowError: join() result is too long for a Python string > > > > Attempt 1 failed. OverflowError: join() result is too long for a Python > string > > Backtrace of previous error: Traceback (innermost last): > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > > 361, in inner_retry > > return fn(self, *args) > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 534, in get > > self.backend._get(remote_filename, local_path) > > File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > > line 110, in _get > > self.client.DownloadResource(entry, local_path.name) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 369, in download_resource > > self._download_file(uri, file_path, **kwargs) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 468, in _download_file > > f.write(self._get_content(uri, **kwargs)) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 451, in _get_content > > return server_response.read() > > File "/usr/lib/python2.7/httplib.py", line 541, in read > > return self._read_chunked(amt) > > File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked > > value.append(self._safe_read(chunk_left)) > > File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read > > chunk = self.fp.read(min(amt, MAXAMOUNT)) > > File "/usr/lib/python2.7/socket.py", line 396, in read > > buf.write(data) > > MemoryError: out of memory > > > > Attempt 2 failed. MemoryError: out of memory > > Backtrace of previous error: Traceback (innermost last): > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > > 361, in inner_retry > > return fn(self, *args) > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 534, in get > > self.backend._get(remote_filename, local_path) > > File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > > line 110, in _get > > self.client.DownloadResource(entry, local_path.name) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 369, in download_resource > > self._download_file(uri, file_path, **kwargs) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 468, in _download_file > > f.write(self._get_content(uri, **kwargs)) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 451, in _get_content > > return server_response.read() > > File "/usr/lib/python2.7/httplib.py", line 541, in read > > return self._read_chunked(amt) > > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > > return ''.join(value) > > OverflowError: join() result is too long for a Python string > > > > Attempt 3 failed. OverflowError: join() result is too long for a Python > string > > Backtrace of previous error: Traceback (innermost last): > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > > 361, in inner_retry > > return fn(self, *args) > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 534, in get > > self.backend._get(remote_filename, local_path) > > File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > > line 110, in _get > > self.client.DownloadResource(entry, local_path.name) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 369, in download_resource > > self._download_file(uri, file_path, **kwargs) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 468, in _download_file > > f.write(self._get_content(uri, **kwargs)) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 451, in _get_content > > return server_response.read() > > File "/usr/lib/python2.7/httplib.py", line 541, in read > > return self._read_chunked(amt) > > File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked > > value.append(self._safe_read(chunk_left)) > > File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read > > chunk = self.fp.read(min(amt, MAXAMOUNT)) > > File "/usr/lib/python2.7/socket.py", line 400, in read > > buf.write(data) > > MemoryError: out of memory > > > > Attempt 4 failed. MemoryError: out of memory > > Backtrace of previous error: Traceback (innermost last): > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > > 361, in inner_retry > > return fn(self, *args) > > File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line > 534, in get > > self.backend._get(remote_filename, local_path) > > File > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", > > line 110, in _get > > self.client.DownloadResource(entry, local_path.name) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 369, in download_resource > > self._download_file(uri, file_path, **kwargs) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 468, in _download_file > > f.write(self._get_content(uri, **kwargs)) > > File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py", > > line 451, in _get_content > > return server_response.read() > > File "/usr/lib/python2.7/httplib.py", line 541, in read > > return self._read_chunked(amt) > > File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked > > return ''.join(value) > > OverflowError: join() result is too long for a Python string > > > > Giving up after 5 attempts. OverflowError: join() result is too long > > for a Python string > > Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c> > > Removing still remembered temporary file > > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1 > > $ > > > > I am running 32-bit linux with 4gb RAM and 4gb swap. > > > > I'll try downloading the backup to local storage as you suggest. > > > > please do.. looks like httplib fails on 2GB files and there isn't much we > can do about that. > > Mike, Ken: any idea? > > ..ede/duply.net > > _______________________________________________ > Mailing list: https://launchpad.net/~duplicity-team > Post to : [email protected] > Unsubscribe : https://launchpad.net/~duplicity-team > More help : https://help.launchpad.net/ListHelp >
_______________________________________________ Mailing list: https://launchpad.net/~duplicity-team Post to : [email protected] Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
