Google cloud works and works well enough indeed. All of them work... but a
non-block technique is capped at 5TB by almost all providers, and getting to
there is complicated and can DOUBLE your storage cost over one month [only] if
you cannot make temporaries without being charged. (looking at you, Wasabi!!).
However either A or B or C for Google specifically...
A) the desired block size must be kept automatically small (it varies but ~40MB
or smaller buffer or so for a 4GB system) ... and each DLE "tape" must be
limited in size
B) the biggest block size can be used to store 5TB objects [max == 512MB] but
the curl buffers will take ~2.5GB and must be hardcoded [currently] in the
build. Its too much for many systems.
C) the biggest block size can be used but google cannot FAIL EVEN ONCE ... or
the cloud upload cannot be restarted and the DLE fails basically. This
doesn't succeed often on the way to 5TB.
D) the biggest block size can be used but Multi-Part must be turned off and a
second DLE and later gets to be very very very slow to add on
Option D has NO limits to backups, but it is what needs the O(log N) check for
single-stored blocks.
This currently does a O(N) check against earlier blocks to check the cloud
storage total after file #1 every time. Verrry slow at only 1000 per xaction.
> -Original Message-
> From: Stefan G. Weichinger
> Sent: Monday, April 4, 2022 1:40 AM
> To: Chris Hassell ; AMANDA users us...@amanda.org>
> Subject: Re: using Google Cloud for virtual tapes
>
> WARNING: This email originated from outside of BETSOL. Do not click links or
> open attachments unless you recognize the sender and know the content is safe.
>
>
> Am 29.03.22 um 15:34 schrieb Chris Hassell:
> > Google Cloud is somewhat difficult because they don't fully the support
> Amazon S3 operations. One cannot upload blocks and "CopyPart" them into a
> larger object. Wasabi and S3 and others can do that.
> >
> > There needs to be a simple overhaul of the "millions of blocks" upload
> technique (non-multipart backups) so that it can be done without O(n) checks
> for
> every DLE.
>
> So is it usable? Or do I have to do some combo of local vtapes and amvaulting
> them into the cloud, maybe?
Confidentiality Notice | The information transmitted by this email is intended
only for the person or entity to which it is addressed. This email may contain
proprietary, business-confidential and/or privileged material. If you are not
the intended recipient of this message, be aware that any use, review,
re-transmission, distribution, reproduction or any action taken in reliance
upon this message is strictly prohibited. If you received this in error, please
contact the sender and delete the material from all computers.
