Hi everybody, We are running wal-e + softlayer's object storage for basebackups and wal archival.
Backups are being pushed successfully to object storage, and they are encrypted with gpg successfully. When I attempt to backup-fetch, wal-e gets to part_0000004 and fails to decrypt it. Here are the relevant bits: (slobj)postgres@oltp0:~$ ./fetch_backups.py * keychain 2.7.1 ~ http://www.funtoo.org * Found existing gpg-agent: 30714 gpg: encrypted with 4096-bit RSA key, ID 0347A438, created 2015-05-05 " <REDACTED>" * keychain 2.7.1 ~ http://www.funtoo.org * Found existing gpg-agent: 30714 gpg: encrypted with 4096-bit RSA key, ID 0347A438, created 2015-05-05 "<REDACTED>" (slobj)postgres@oltp0:~$ ls wale_manual_decrypt/ base part_00000004.tar part_00000004.tar.lzo part_00000220.tar part_00000220.tar.lzo What's interesting is wal-e looks like it decrypts some of the part_00000*, up to until the 4th, when it hits part_0000004.tar.lzo it starts throwing the gpg errors. When I run postgres@oltp0:~$ eval $( keychain --eval --agents gpg ) && env && envdir /etc/wale.d/env/ wale_venv/bin/wal-e backup-fetch backup/ LATEST It looks like it decrypts some of the stuff postgres@oltp0:~$ ls backup/base/21562/ 114659 11937_fsm 11945 12093 22857.3 23072 23079 23479 23638 23999 26775 26779 26798_vm 26947 27185_fsm 27417_fsm 27460.11 27520 37652.2 37668 64764 64890 65064 65212 65369 65657 82953_vm postgres@oltp0:~$ ls backup/base/21562/ 114659 11937_fsm 11945 12093 22857.3 23072 23079 23479 23638 23999 26775 26779 26798_vm 26947 27185_fsm 27417_fsm 27460.11 27520 37652.2 37668 64764 64890 65064 65212 65369 65657 82953_vm postgres@oltp0:~$ ls backup/base/ 21562/ 76764/ pgsql_tmp/ postgres@oltp0:~$ ls backup/base/76764/ 11875 11881 11892 11897_fsm 11909 11916_vm 11929 11937_fsm 11943 11949 11956 11965 11970_vm 11983 11997 12009 12014 12050_vm 12057 12060 12069 12077 12095 12103_vm 12113_vm 12118_vm 12132 76774_fsm 11875_vm 11883 11893_fsm 11901 11915 11923 11929_vm 11937_vm 11944 11949_fsm 11958 11965_vm 11973 11985 12002 12010 12046_fsm 12052 12057_fsm 12061 12072 12084 12097 12108_vm 12115 12125 12135 76774_vm 11877 11884 11895 11904_vm 11916 11925 11933_fsm 11939 11945 11949_vm 11958_vm 11967 11975 11987_vm 12002_vm 12011_fsm 12048 12053 12057_vm 12061_fsm 12075 12089 12099 12110 12117 12128 12137 76777 11880 11891 11896 11908 11916_fsm 11927 11937 11941 11948 11951 11961 11970 11982 11993_vm 12006_fsm 12011_vm 12050_fsm 12056 12059 12065_fsm 12076 12093 12103_fsm 12113 12118 12128_vm 76774 76779 but then fails gpg: [don't know]: invalid packet (ctb=63) gpg: mdc_packet with invalid encoding gpg: decryption failed: invalid packet gpg: [don't know]: invalid packet (ctb=61) gpg: block_filter: pending bytes! gpg: [don't know]: invalid packet (ctb=56) gpg: mdc_packet with invalid encoding gpg: decryption failed: invalid packet gpg: WARNING: multiple plaintexts seen gpg: handle plaintext failed: unexpected data gpg: onepass_sig with unknown version 183 lzop: Inappropriate ioctl for device: <stdin> lzop: Inappropriate ioctl for device: <stdin> ################# Fetch script #!/usr/bin/env python import os try: import object_storage except ImportError: print "do virtualenv slobj && . slobj/bin/activate && pip install softlayer-object-storage" sl = object_storage.get_client( os.getenv('softlayer_api_username'), os.getenv('softlayer_api_key'), datacenter=os.getenv('softlayer_api_datacenter') ) build_dir = os.path.join(os.getenv('HOME'), 'wale_manual_decrypt') try: shutil.rmtree(build_dir) except OSError: pass os.mkdir(build_dir) # grab the backup that was failing [ x.save_to_filename(os.path.join(build_dir, os.path.basename(x.name))) for x in sl[<REDACTED>].objects(6) if '004.tar.lzo' in x.path in x.path ] # grab the most recent backup [ x.save_to_filename(os.path.join(build_dir, os.path.basename(x.name))) for x in sl[<REDACTED>].objects() if '220.tar.lzo' in x.path and '1570' in x.path ] fail_backup = '' recent_backup = '' for f in os.listdir(build_dir): if f.endswith('004.tar.lzo') or f.endswith('220.tar.lzo'): cmd = '''eval $( keychain --eval --agents gpg ) && gpg --output {0} --decrypt {1} && cd {2} && tar -xf {3}'''.format( os.path.join(build_dir, f[:-4]), os.path.join(build_dir, f), build_dir, f[:-4]) os.system(cmd) ############### backup output * keychain 2.7.1 ~ http://www.funtoo.org * Found existing gpg-agent: 5653 GPG_AGENT_INFO=/tmp/gpg-8ocgCZ/S.gpg-agent:5653: _=/usr/bin/env gpg: [don't know]: invalid packet (ctb=63) gpg: mdc_packet with invalid encoding gpg: decryption failed: invalid packet gpg: [don't know]: invalid packet (ctb=61) gpg: block_filter: pending bytes! gpg: [don't know]: invalid packet (ctb=56) gpg: mdc_packet with invalid encoding gpg: decryption failed: invalid packet gpg: WARNING: multiple plaintexts seen gpg: handle plaintext failed: unexpected data gpg: onepass_sig with unknown version 183 lzop: Inappropriate ioctl for device: <stdin> lzop: Inappropriate ioctl for device: <stdin> ^C gpg: gpgInterrupt: caught ... exiting Interrupt caught ... exiting ^CTraceback (most recent call last): File "wale_venv/bin/wal-e", line 9, in <module> load_entry_point('wal-e==0.8.1', 'console_scripts', 'wal-e')() File "/var/lib/postgresql/wale_venv/local/lib/python2.7/site-packages/wal_e/cmd.py", line 545, in main pool_size=args.pool_size) postgres@oltp0:~$ ls backup/base/21562/ 114659 11937_fsm 11945 12093 22857.3 23072 23079 23479 23638 23999 26775 26779 26798_vm 26947 27185_fsm 27417_fsm 27460.11 27520 37652.2 37668 64764 64890 65064 65212 65369 65657 82953_vm postgres@oltp0:~$ ls backup/base/21562/ 114659 11937_fsm 11945 12093 22857.3 23072 23079 23479 23638 23999 26775 26779 26798_vm 26947 27185_fsm 27417_fsm 27460.11 27520 37652.2 37668 64764 64890 65064 65212 65369 65657 82953_vm postgres@oltp0:~$ ls backup/base/ 21562/ 76764/ pgsql_tmp/ postgres@oltp0:~$ ls backup/base/76764/ 11875 11881 11892 11897_fsm 11909 11916_vm 11929 11937_fsm 11943 11949 11956 11965 11970_vm 11983 11997 12009 12014 12050_vm 12057 12060 12069 12077 12095 12103_vm 12113_vm 12118_vm 12132 76774_fsm 11875_vm 11883 11893_fsm 11901 11915 11923 11929_vm 11937_vm 11944 11949_fsm 11958 11965_vm 11973 11985 12002 12010 12046_fsm 12052 12057_fsm 12061 12072 12084 12097 12108_vm 12115 12125 12135 76774_vm 11877 11884 11895 11904_vm 11916 11925 11933_fsm 11939 11945 11949_vm 11958_vm 11967 11975 11987_vm 12002_vm 12011_fsm 12048 12053 12057_vm 12061_fsm 12075 12089 12099 12110 12117 12128 12137 76777 11880 11891 11896 11908 11916_fsm 11927 11937 11941 11948 11951 11961 11970 11982 11993_vm 12006_fsm 12011_vm 12050_fsm 12056 12059 12065_fsm 12076 12093 12103_fsm 12113 12118 12128_vm 76774 76779 ############ version info postgres@oltp0:~$ uname -a Linux 3.13.0-61-generic #100-Ubuntu SMP Wed Jul 29 11:21:34 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux postgres@oltp0:~$ wale_venv/bin/wal-e version 0.8.1 postgres@oltp0:~$ wale_venv/bin/pip freeze Babel==2.0 argparse==1.2.1 azure==0.11.1 boto==2.38.0 cffi==1.2.1 cryptography==1.0 debtcollector==0.6.0 enum34==1.0.4 futures==3.0.3 gevent==1.0.2 greenlet==0.4.7 idna==2.0 ipaddress==1.0.14 iso8601==0.1.10 monotonic==0.2 msgpack-python==0.4.6 ndg-httpsclient==0.4.0 netaddr==0.7.15 netifaces==0.10.4 oslo.config==2.1.0 oslo.i18n==2.2.0 oslo.serialization==1.7.0 oslo.utils==2.1.0 pbr==1.3.0 prettytable==0.7.2 pyOpenSSL==0.15.1 pyasn1==0.1.8 pycparser==2.14 python-dateutil==2.4.2 python-keystoneclient==1.6.0 python-swiftclient==2.5.0 pytz==2015.4 requests==2.7.0 six==1.9.0 stevedore==1.6.0 wal-e==0.8.1 wrapt==1.10.5 wsgiref==0.1.2 postgres@oltp0:~$ pg_config --version PostgreSQL 9.4.4 -- You received this message because you are subscribed to the Google Groups "wal-e" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
