Re: [Bacula-users] Anyone using AWS tape gateway or other similar service?

2017-08-28 Thread Phil Stracchino 
On 08/28/17 07:50, Josh Fisher wrote:
> I have considered a cloud solution as an off-site storage, but abandoned
> it for now for due to that very reason. Cloud storage is simply not yet
> fast enough, or I should say that the Internet is not yet fast enough.
> What if the restore would require multiple TB? What if even incremental
> backups required large transfers, say for some data collection services,
> busy mail servers, etc.? At some point the Bacula service would require
> a dedicated Internet service or else dedicated bandwidth. Where I live,
> that would make cloud storage the most expensive media.
[...]
> Nevertheless, I do believe cloud storage is the future for backups, or
> at least as off-site backup storage. It is just a matter of bandwidth
> becoming faster and cheaper.


I simply do not see Internet service — at least in the US — increasing
in speed faster than robust, non-volatile local storage increases in
capacity-per-unit-cost.  Most of the US does not HAVE Internet service
that meets the FCC's working definition of broadband, most non-business
customers who *do* have access to broadband have only one choice, and at
that ISPs have dragged their heels for decades on delivering.  The
telcos accepted vast amounts of government funding to deliver high-speed
Internet service to US consumers, and spent it on building out cellular
infrastructure instead because they could make more money off that.
Verizon finally started rolling out fiber service to the home in 2005,
and then five years after that they basically stopped expansion and sold
off all of their northern New England physical plant to Fairpoint
Communications because they were making more money off cellular.
Fairpoint almost collapsed absorbing it, hasn't run a single mile of
fiber since, promised to have broadband throughout New Hampshire within
five years, and still hasn't delivered.  Fairpoint has abandoned two
infrastructure plans and has finally fallen back on ADSL over copper
phone lines, which is laughable as a broadband solution.  You may be
able to get a couple of megabits down and half a megabit up if you're
close enough to your nearest telco central office.

I have small-business internet service through my local cable carrier,
the only broadband choice available to me short of having them run a
fiber link to me all the way from Boston, and with 4.5 megabits upload
it would take me 27 days to run a full backup to the cloud, saturating
my uplink the entire time.  Then I'd have three to four days of usable
Internet before the next full backup.  And the odds are, the faster a
connection you can afford, the more data you have to be backed up, so
there's limits to how well throwing more money at your Internet service
scales.

Typical broadband internet - in the US at least - is going to have to
become at least two full orders of magnitude faster before it is viable
as a backup solution for anything significantly more than local settings
and preferences, your email, and the contents of a typical cluttered
desktop.  If you need *FULL* backups and your only choice is to back up
to the cloud, you're better off doing it by some form of
continuous-incremental scheme such as nightly rsyncs.  The US likes to
think of itself as the world's technological leader, but in terms of
broadband availability nationwide we're almost on a par with North Korea.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Anyone using AWS tape gateway or other similar service?

2017-08-28 Thread Luke Salsich 
I agree Josh. As we review this current internet speed limitation more, I
think we will use AWS cloud for servers with small to moderate data size.
For example, a Linux server which manages a web app and has a total disk
size of under 50 GB will go to the cloud.

Servers with storage which requires backup and is over 50 GB we will store
locally.

I'm also going to look into backing up one server using two jobs and two
filesets - one job for large, local storage and one job for small, critical
data for the cloud. I assume this is workable, but I haven't tested the
functionality yet or worked through the logic thoroughly.

The reason I want to look into this is two-job approach is:

1. Cloud storage through AWS Storage Gateway is incredibly cheap, reliable
and easy to scale
2. We need to have critical files stored off-site in case of a local
disaster


---
Luke Salsich

On Mon, Aug 28, 2017 at 7:50 AM, Josh Fisher  wrote:

>
> On 8/25/2017 1:49 PM, Luke Salsich wrote:
>
>
> One remaining concern which I have (not related to Bacula at all) is the
> time it would take to conduct a complete system restore due to needing to
> transfer the data from AWS to our local. Even with their fast connection, a
> 150 GB transfer could take about 24 hours.
>
> Any thoughts on this or other cloud-based tape backup solutions using
> Bacula?
>
>
> I have considered a cloud solution as an off-site storage, but abandoned
> it for now for due to that very reason. Cloud storage is simply not yet
> fast enough, or I should say that the Internet is not yet fast enough. What
> if the restore would require multiple TB? What if even incremental backups
> required large transfers, say for some data collection services, busy mail
> servers, etc.? At some point the Bacula service would require a dedicated
> Internet service or else dedicated bandwidth. Where I live, that would make
> cloud storage the most expensive media.
>
> Another concern is disaster recovery. Following a disaster, while Internet
> service in the area may be available fairly quickly, full bandwidth to an
> area may not be restored for quite some time, causing a restore to be
> problematic. With physical media, at least systems can be restored to make
> use of what bandwidth is available.
>
> Nevertheless, I do believe cloud storage is the future for backups, or at
> least as off-site backup storage. It is just a matter of bandwidth becoming
> faster and cheaper.
>
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Anyone using AWS tape gateway or other similar service?

2017-08-28 Thread Josh Fisher 


On 8/25/2017 1:49 PM, Luke Salsich wrote:


One remaining concern which I have (not related to Bacula at all) is 
the time it would take to conduct a complete system restore due to 
needing to transfer the data from AWS to our local. Even with their 
fast connection, a 150 GB transfer could take about 24 hours.


Any thoughts on this or other cloud-based tape backup solutions using 
Bacula?


I have considered a cloud solution as an off-site storage, but abandoned 
it for now for due to that very reason. Cloud storage is simply not yet 
fast enough, or I should say that the Internet is not yet fast enough. 
What if the restore would require multiple TB? What if even incremental 
backups required large transfers, say for some data collection services, 
busy mail servers, etc.? At some point the Bacula service would require 
a dedicated Internet service or else dedicated bandwidth. Where I live, 
that would make cloud storage the most expensive media.


Another concern is disaster recovery. Following a disaster, while 
Internet service in the area may be available fairly quickly, full 
bandwidth to an area may not be restored for quite some time, causing a 
restore to be problematic. With physical media, at least systems can be 
restored to make use of what bandwidth is available.


Nevertheless, I do believe cloud storage is the future for backups, or 
at least as off-site backup storage. It is just a matter of bandwidth 
becoming faster and cheaper.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Anyone using AWS tape gateway or other similar service?

2017-08-25 Thread Luke Salsich 
Hi All,

I've posted a couple of questions here in the last few months about our
12-month test of switching Bacula to using virtual tapes, drives and
changers through the AWS Storage Gateway.

We are about 10 months into our test and after overcoming a bunch of
hurdles, things seem to be going very well.

However, I wanted to ask if anyone else has or is using the AWS Storage
Gateway (or a similar, cloud setup) to manage backups? What concerns do you
have? What issues have you run into?

I ask because I value this community's expertise and also because there are
very few others who have publically documented a similar test in their
infrastructure.

One remaining concern which I have (not related to Bacula at all) is the
time it would take to conduct a complete system restore due to needing to
transfer the data from AWS to our local. Even with their fast connection, a
150 GB transfer could take about 24 hours.

Any thoughts on this or other cloud-based tape backup solutions using
Bacula?

---
Luke Salsich
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users