> Il giorno 29 ago 2017, alle ore 14:14, Jean-Louis Martineau
> <jmartin...@carbonite.com> ha scritto:
>
> Are you using the same bucket for another amanda configuration?
As far as I can see, no. But then again, I could very well be missing something
obvious
/etc/amanda/amanda.conf:
> infofile "/var/lib/amanda/state/curinfo"
> indexdir "/var/lib/amanda/state/index"
> dumpuser "backup"
> mailto “<xxx>"
>
> define changer s3 {
> tapedev
> "chg-multi:s3:onebip-backups-admin/slot-bb2e601e63b3408bb6a865e442a28366-{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}"
> changerfile "/var/lib/amanda/state/s3-statefile"
> device-property "NB_THREADS_BACKUP" "8"
> device-property "NB_THREADS_RECOVERY" "8"
> device_property "S3_ACCESS_KEY” “<>"
> device_property "S3_SECRET_KEY” “<>"
> device_property "S3_BUCKET_LOCATION" "eu-west-1"
> device_property "S3_SSL" "YES"
> device_property "BLOCK_SIZE" "10 megabytes"
> }
>
> tpchanger "s3"
> tapetype S3
>
> define tapetype S3 {
> comment "S3 Bucket"
> length 10240 gigabytes
> }
/etc/amanda/zfs-dataset/amanda.conf:
> includefile "../amanda.conf”
>
> org "zfs-dataset"
> logdir "/var/log/zfs-dataset"
>
> define application-tool zfs-dataset-app {
> comment "amzfs-sendrecv"
> plugin "amzfs-sendrecv"
> property "ZFS-PATH" "/sbin/zfs"
> property "PFEXEC-PATH" "/usr/bin/sudo"
> property "PFEXEC" "YES"
> }
>
> define dumptype zfs-dataset-dump {
> program "APPLICATION"
> application "zfs-dataset-app"
> auth "ssh"
> ssh_keys "/var/backups/.ssh/id_rsa"
> }
/etc/amanda/zfs-dataset/zfs-dataset/disklist:
> ci-master rpool/jenkins zfs-dataset-dump
> backup-server rpool/amanda-conf zfs-dataset-dump
> backup-server rpool/amanda-home zfs-dataset-dump
> backup-server rpool/amanda-state zfs-dataset-dump
> vpn-server rpool/openvpn zfs-dataset-dump
root@backup-server:/etc/amanda# cat zfs-dataset/tapelist:
> 20170829000017 slot-13 reuse BLOCKSIZE:10240
> 20170828000017 slot-12 reuse BLOCKSIZE:10240
> 20170827000017 slot-11 reuse BLOCKSIZE:10240
> 20170826000017 slot-10 reuse BLOCKSIZE:10240
> 20170825000017 slot-9 reuse BLOCKSIZE:10240
> 20170824000017 slot-8 reuse BLOCKSIZE:10240
> 20170823000017 slot-7 reuse BLOCKSIZE:10240
> 20170822000017 slot-6 reuse BLOCKSIZE:10240
> 20170821000017 slot-5 reuse BLOCKSIZE:10240
> 20170820000017 slot-4 reuse BLOCKSIZE:10240
> 20170819000017 slot-3 reuse BLOCKSIZE:10240
> 20170818000017 slot-2 reuse BLOCKSIZE:10240
> 20170817000017 slot-1 reuse BLOCKSIZE:10240
> 20170816000017 slot-15 reuse BLOCKSIZE:10240
> 20170815000017 slot-14 reuse BLOCKSIZE:1024
> Do you have a lifecycle policy on the bucket?
>
> After a backup, you can do: amcheckdump CONF
> to verify amanda can read the backup.
>
> You can also get the list of all s3 objects and check if something was
> removed.
>
> Jean-Louis
>
>
> On 29/08/17 08:00 AM, Giorgio Valoti wrote:
>>
>> > Il giorno 29 ago 2017, alle ore 13:55, Jean-Louis Martineau
>> > <jmartin...@carbonite.com> ha scritto:
>> >
>> > The list of files with the 'slot-bb2e601e63b3408bb6a865e442a28366-6'
>> > prefix (slot-6) show there is no data in the slot, only the tape label.
>> > There is data only on slot-10,11,12 and 13.
>> > I have no idea why there is no data in the others slots.
>> > I assume the data was there after the backup, but I don't how they got
>> > erased.
>>
>> As far as I can tell, nothing touches this bucket except Amanda. What should
>> I check or keep checked to avoid issues like this? Do you think should I
>> change / check something in my conf?
>>
>> >
>> > Jean-Louis
>> >
>> > On 29/08/17 07:31 AM, Giorgio Valoti wrote:
>> >>
>> >> > Il giorno 29 ago 2017, alle ore 12:36, Jean-Louis Martineau
>> >> > <jmartin...@carbonite.com> ha scritto:
>> >> >
>> >> > This error often means that some s3 objects are missing.
>> >> >
>> >> > Do some of the data was restored?
>> >> > Can you post the amfetchdump debug file?
>> >>
>> >> Here it is:
>> >>
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 ruid 34
>> >> euid 34 version 3.3.6: start at Tue Aug 29 08:27:19 2017
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Arguments: -p -a
>> >> zfs-dataset ci-master rpool/jenkins 20170822000017
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 ruid 34
>> >> euid 34 version 3.3.6: rename at Tue Aug 29 08:27:19 2017
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Using state file:
>> >> /var/lib/amanda/state/s3-statefile
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump:
>> >> Amanda::Recovery::Clerk: loading volume 'slot-6'
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: find_volume labeled
>> >> 'slot-6'
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: parse_inventory:
>> >> load slot 6 with label 'slot-6'
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: S3 driver using
>> >> bucket 'onebip-backups-admin', prefix
>> >> 'slot-bb2e601e63b3408bb6a865e442a28366-6'
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: curl version:
>> >> libcurl/7.47.0 OpenSSL/1.0.2g zlib/1.2.8 libidn/1.32 librtmp/2.3
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: curl compiled for
>> >> OPENSSL
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Create 8 threads
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: GET
>> >> https://onebip-backups-admin.s3.amazonaws.com/slot-bb2e601e63b3408bb6a865e442a28366-6f00000003-filestartfailed
>> >> with 404/NoSuchKey
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: GET
>> >> https://onebip-backups-admin.s3.amazonaws.com/slot-bb2e601e63b3408bb6a865e442a28366-6f00000002-filestartfailed
>> >> with 404/NoSuchKey
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Device
>> >> s3:onebip-backups-admin/slot-bb2e601e63b3408bb6a865e442a28366-6 error =
>> >> 'Attempt to read past tape-end file'
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: FAILURE: Attempt to
>> >> read past tape-end file
>> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 finish
>> >> time Tue Aug 29 08:27:19 2017
>> >>
>> >>
>> >>
>> >> > Can you get a list of all objects in the S3 bucket?
>> >>
>> >>
>> >> Here it is. I have scraped the web page and yes the key is missing but I
>> >> don’t know why. I mean, I receive the reports every day and they seemed
>> >> OK to me. Let me know if you need my conf, as well, thank you.
>> >>
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000001-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000001-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000002-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000002-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000001.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000002.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000003.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000004.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000005.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000006.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000007.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000008.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000009.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000a.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000b.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000c.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000d.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000e.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000f.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000010.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000011.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000012.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000013.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000014.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000015.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000016.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000017.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000018.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000019.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000001a.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000004-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000004-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000005-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000005-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-10special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000001-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000001-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000002-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000002-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000003-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000003-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000004-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000004-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000005-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000005-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-11special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000001-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000001-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000002-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000002-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000003-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000003-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000004-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000004-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000005-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000005-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-12special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000001-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000001-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000002-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000002-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000001.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000002.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000003.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000004.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000005.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000006.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000007.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000008.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000009.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000a.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000b.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000c.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000d.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000e.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000f.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000010.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000011.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000012.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000013.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000014.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000015.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000016.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000017.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000018.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000019.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001a.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001b.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001c.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001d.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001e.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001f.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000020.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000021.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000022.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000004-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000004-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000005-b0000000000000000.data
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000005-filestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-13special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-14special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-15special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-1special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-2special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-3special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-4special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-5special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-6special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-7special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-8special-tapestart
>> >> slot-bb2e601e63b3408bb6a865e442a28366-9special-tapestart
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> >
>> >> > Jean-Louis
>> >> >
>> >> > On 29/08/17 05:22 AM, Giorgio Valoti wrote:
>> >> > > Hi all, I’m testing the restore process with Amanda and I’m getting
>> >> > > this error with amfetchdump:
>> >> > >
>> >> > > amfetchdump -p -a zfs-dataset ci-master rpool/jenkins 20170822000017
>> >> > >
>> >> > >> 1 volume(s) needed for restoration
>> >> > >> The following volumes are needed: slot-6
>> >> > >> ERROR: Attempt to read past tape-end file
>> >> > > And here is the list of the relevant backups:
>> >> > >
>> >> > >> backup@backup-server:~$ amadmin zfs-dataset find --sort D ci-master
>> >> > >> rpool/jenkins
>> >> > >>
>> >> > >> date host disk lv tape or file file part status
>> >> > >> 2017-08-29 00:00:17 ci-master rpool/jenkins 2 slot-13 3 1/1 OK
>> >> > >> 2017-08-28 00:00:17 ci-master rpool/jenkins 2 slot-12 1 1/1 OK
>> >> > >> 2017-08-27 00:00:17 ci-master rpool/jenkins 2 slot-11 1 1/1 OK
>> >> > >> 2017-08-26 00:00:17 ci-master rpool/jenkins 1 slot-10 3 1/1 OK
>> >> > >> 2017-08-25 00:00:17 ci-master rpool/jenkins 1 slot-9 3 1/1 OK
>> >> > >> 2017-08-24 00:00:17 ci-master rpool/jenkins 1 slot-8 3 1/1 OK
>> >> > >> 2017-08-23 00:00:17 ci-master rpool/jenkins 1 slot-7 3 1/1 OK
>> >> > >> 2017-08-22 00:00:17 ci-master rpool/jenkins 0 slot-6 3 1/1 OK
>> >> > >> 2017-08-21 00:00:17 ci-master rpool/jenkins 2 slot-5 1 1/1 OK
>> >> > >> 2017-08-20 00:00:17 ci-master rpool/jenkins 2 slot-4 1 1/1 OK
>> >> > >> 2017-08-19 00:00:17 ci-master rpool/jenkins 2 slot-3 1 1/1 OK
>> >> > >> 2017-08-18 00:00:17 ci-master rpool/jenkins 2 slot-2 1 1/1 OK
>> >> > >> 2017-08-17 00:00:17 ci-master rpool/jenkins 1 slot-1 3 1/1 OK
>> >> > >> 2017-08-16 00:00:17 ci-master rpool/jenkins 1 slot-15 1 1/1 OK
>> >> > >> 2017-08-15 00:00:17 ci-master rpool/jenkins 1 slot-14 1 1/1 OK
>> >> > > In other words, I’m trying to fetch the level 0 backup. Also, it
>> >> > > should be noted that I’m using “tapes” with other Amanda clients as
>> >> > > well.:
>> >> > >
>> >> > >> backup@backup-server:~$ amadmin zfs-dataset find --sort D | grep
>> >> > >> slot-6
>> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-conf 1 slot-6 5 1/1 OK
>> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-state 1 slot-6 4 1/1
>> >> > >> OK
>> >> > >> 2017-08-22 00:00:17 ci-master rpool/jenkins 0 slot-6 3 1/1 OK
>> >> > >> 2017-08-22 00:00:17 vpn-server rpool/openvpn 1 slot-6 2 1/1 OK
>> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-home 1 slot-6 1 1/1 OK
>> >> > > I’m sure I’m missing something here, because this is a real bummer.
>> >> > > Anyone on this list willing to help me, please?
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Giorgio Valoti
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> > Disclaimer
>> >> >
>> >> > This message is the property of CARBONITE, INC. and may contain
>> >> > confidential or privileged information.
>> >> >
>> >> > If this message has been delivered to you by mistake, then do not copy
>> >> > or deliver this message to anyone. Instead, destroy it and notify me by
>> >> > reply e-mail.
>> >> >
>> >>
>> >>
>> >> --
>> >> Giorgio Valoti
>> >
>> >
>> >
>> >
>> > Disclaimer
>> >
>> > This message is the property of CARBONITE, INC. and may contain
>> > confidential or privileged information.
>> >
>> > If this message has been delivered to you by mistake, then do not copy or
>> > deliver this message to anyone. Instead, destroy it and notify me by reply
>> > e-mail.
>> >
>
>
>
>
> Disclaimer
>
> This message is the property of CARBONITE, INC. and may contain confidential
> or privileged information.
>
> If this message has been delivered to you by mistake, then do not copy or
> deliver this message to anyone. Instead, destroy it and notify me by reply
> e-mail.
>
