Re: [Openstack-operators] backup to object store - tool recommendations

[Blair Bethwaite]

Hi Saverio. Looks like we will have that issue within a week or two once we
finally upgrade our rgw (lagging behind the rest of our cluster). I will
try logging​ this as a priority with Red Hat support.

On 29 Mar. 2017 18:39, "Saverio Proto"  wrote:

> Hello all,
>
> we use rclone a lot, and we are happy with it.
>
> the real problem I would say is that a lot of these tools use the
> latest AWS4 signature.
>
> AFAIK the radosgw with Ceph Jewel and Openstack keystone integration
> supports only AWS2 signature because of this bug:
> http://tracker.ceph.com/issues/19056
>
> is anyone else hitting this ?
>
> Saverio
>
> 2017-03-27 22:11 GMT+02:00 John Dickinson :
> >
> >
> > On 27 Mar 2017, at 4:39, Blair Bethwaite wrote:
> >
> >> Hi all,
> >>
> >> Does anyone have any recommendations for good tools to perform
> >> file-system/tree backups and restores to/from a (Ceph RGW-based)
> >> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
> >> commercial options please.
> >>
> >> I'm interested in:
> >> 1) tools known to work or not work at all for a basic file-based data
> backup
> >
> > There's a bunch of backup tools that will work with the Swift API and/or
> the S3 API.
> >
> > Veritas, Commvault, Trilio, and CloudBerry all work. There's other
> companies too that can back specific stuff up to Swift (e.g. Percona with
> MySQL).
> >
> > (The above list taken from https://www.swiftstack.com/solutions/backup
> [my employer] because it's the first linkable place I knew of to answer
> your question.)
> >
> >
> > --John
> >
> >
> >
> >
> >>
> >> Plus these extras:
> >> 2) preserves/restores correct file metadata (e.g. owner, group, acls
> etc)
> >> 3) preserves/restores xattrs
> >> 4) backs up empty directories and files
> >> 5) supports some sort of snapshot/versioning/differential
> >> functionality, i.e., will keep a copy or diff or last N versions of a
> >> file or whole backup set, e.g., so that one can restore yesterday's
> >> file/s or last week's but not have to keep two full copies to achieve
> >> it
> >> 6) is readily able to restore individual files
> >> 7) can encrypt/decrypt client side
> >> 8) anything else I should be considering?
> >>
> >> --
> >> Cheers,
> >> ~Blairo
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Saverio Proto]

Hello all,

we use rclone a lot, and we are happy with it.

the real problem I would say is that a lot of these tools use the
latest AWS4 signature.

AFAIK the radosgw with Ceph Jewel and Openstack keystone integration
supports only AWS2 signature because of this bug:
http://tracker.ceph.com/issues/19056

is anyone else hitting this ?

Saverio

2017-03-27 22:11 GMT+02:00 John Dickinson :
>
>
> On 27 Mar 2017, at 4:39, Blair Bethwaite wrote:
>
>> Hi all,
>>
>> Does anyone have any recommendations for good tools to perform
>> file-system/tree backups and restores to/from a (Ceph RGW-based)
>> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
>> commercial options please.
>>
>> I'm interested in:
>> 1) tools known to work or not work at all for a basic file-based data backup
>
> There's a bunch of backup tools that will work with the Swift API and/or the 
> S3 API.
>
> Veritas, Commvault, Trilio, and CloudBerry all work. There's other companies 
> too that can back specific stuff up to Swift (e.g. Percona with MySQL).
>
> (The above list taken from https://www.swiftstack.com/solutions/backup [my 
> employer] because it's the first linkable place I knew of to answer your 
> question.)
>
>
> --John
>
>
>
>
>>
>> Plus these extras:
>> 2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
>> 3) preserves/restores xattrs
>> 4) backs up empty directories and files
>> 5) supports some sort of snapshot/versioning/differential
>> functionality, i.e., will keep a copy or diff or last N versions of a
>> file or whole backup set, e.g., so that one can restore yesterday's
>> file/s or last week's but not have to keep two full copies to achieve
>> it
>> 6) is readily able to restore individual files
>> 7) can encrypt/decrypt client side
>> 8) anything else I should be considering?
>>
>> --
>> Cheers,
>> ~Blairo
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[John Dickinson]

On 27 Mar 2017, at 4:39, Blair Bethwaite wrote:

> Hi all,
>
> Does anyone have any recommendations for good tools to perform
> file-system/tree backups and restores to/from a (Ceph RGW-based)
> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
> commercial options please.
>
> I'm interested in:
> 1) tools known to work or not work at all for a basic file-based data backup

There's a bunch of backup tools that will work with the Swift API and/or the S3 
API.

Veritas, Commvault, Trilio, and CloudBerry all work. There's other companies 
too that can back specific stuff up to Swift (e.g. Percona with MySQL).

(The above list taken from https://www.swiftstack.com/solutions/backup [my 
employer] because it's the first linkable place I knew of to answer your 
question.)


--John




>
> Plus these extras:
> 2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
> 3) preserves/restores xattrs
> 4) backs up empty directories and files
> 5) supports some sort of snapshot/versioning/differential
> functionality, i.e., will keep a copy or diff or last N versions of a
> file or whole backup set, e.g., so that one can restore yesterday's
> file/s or last week's but not have to keep two full copies to achieve
> it
> 6) is readily able to restore individual files
> 7) can encrypt/decrypt client side
> 8) anything else I should be considering?
>
> -- 
> Cheers,
> ~Blairo
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


signature.asc
Description: OpenPGP digital signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Joe Topjian]

We use rclone quite a bit. It works great and has a wealth of features:

http://rclone.org/


On Mon, Mar 27, 2017 at 7:50 AM, Nick Jones 
wrote:

> On 27 Mar 2017, at 12:59, Marcus Furlong  wrote:
>
>
> On 27 March 2017 at 22:39, Blair Bethwaite 
> wrote:
>
> Hi all,
>
> Does anyone have any recommendations for good tools to perform
> file-system/tree backups and restores to/from a (Ceph RGW-based)
> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
> commercial options please.
>
>
> [..]
>
>
> I've used duplicity before and it seems to provide most of the
> features you are looking for (not sure about xattrs though):
>
>   http://duplicity.nongnu.org/
>
> S3 and Swift are both supported.
>
>
> +1 for Duplicity - we’ve used it with a degree of success for backups of
> various internal systems.
>
> I can also recommend CloudBerry Backup: https://www.
> cloudberrylab.com/backup.aspx
>
> And for personal stuff on MacOS and Windows Arq is great:
> https://www.arqbackup.com
>
> Cheers.
>
> —
>
> -Nick
>
> DataCentred Limited registered in England and Wales no. 05611763
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Nick Jones]

On 27 Mar 2017, at 12:59, Marcus Furlong  wrote:
> 
> On 27 March 2017 at 22:39, Blair Bethwaite  > wrote:
>> Hi all,
>> 
>> Does anyone have any recommendations for good tools to perform
>> file-system/tree backups and restores to/from a (Ceph RGW-based)
>> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
>> commercial options please.

[..]

> 
> I've used duplicity before and it seems to provide most of the
> features you are looking for (not sure about xattrs though):
> 
>   http://duplicity.nongnu.org/ 
> 
> S3 and Swift are both supported.

+1 for Duplicity - we’ve used it with a degree of success for backups of 
various internal systems.

I can also recommend CloudBerry Backup: 
https://www.cloudberrylab.com/backup.aspx

And for personal stuff on MacOS and Windows Arq is great: 
https://www.arqbackup.com

Cheers.

— 

-Nick
-- 
DataCentred Limited registered in England and Wales no. 05611763
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Marcus Furlong]

On 27 March 2017 at 22:39, Blair Bethwaite  wrote:
> Hi all,
>
> Does anyone have any recommendations for good tools to perform
> file-system/tree backups and restores to/from a (Ceph RGW-based)
> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
> commercial options please.
>
> I'm interested in:
> 1) tools known to work or not work at all for a basic file-based data backup
>
> Plus these extras:
> 2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
> 3) preserves/restores xattrs
> 4) backs up empty directories and files
> 5) supports some sort of snapshot/versioning/differential
> functionality, i.e., will keep a copy or diff or last N versions of a
> file or whole backup set, e.g., so that one can restore yesterday's
> file/s or last week's but not have to keep two full copies to achieve
> it
> 6) is readily able to restore individual files
> 7) can encrypt/decrypt client side
> 8) anything else I should be considering?

I've used duplicity before and it seems to provide most of the
features you are looking for (not sure about xattrs though):

   http://duplicity.nongnu.org/

S3 and Swift are both supported.

There are also a bunch of duplicity frontends available; e.g. duply,
deja-dup, duplicati, etc.

Cheers,
Marcus.

-- 
Marcus Furlong

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] backup to object store - tool recommendations

[Blair Bethwaite]

Hi all,

Does anyone have any recommendations for good tools to perform
file-system/tree backups and restores to/from a (Ceph RGW-based)
object store (Swift or S3 APIs)? Happy to hear about both FOSS and
commercial options please.

I'm interested in:
1) tools known to work or not work at all for a basic file-based data backup

Plus these extras:
2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
3) preserves/restores xattrs
4) backs up empty directories and files
5) supports some sort of snapshot/versioning/differential
functionality, i.e., will keep a copy or diff or last N versions of a
file or whole backup set, e.g., so that one can restore yesterday's
file/s or last week's but not have to keep two full copies to achieve
it
6) is readily able to restore individual files
7) can encrypt/decrypt client side
8) anything else I should be considering?

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators