Re: rsync remote raw block device with --inplace

[Gionatan Danti via rsync]

Il 02-01-2019 21:16 Steve Newcomb via rsync ha scritto:

To summarize: while diskrsync is looking like the most suitable
solution known to us, it would still be more convenient, gentler,
kinder, and wiser if the ability to transfer raw block device content
were added to rsync.


Can I suggest you to try blocksync [1]?
I'm using the git version in production without problems, but (as 
always) you mileage may vary.


[1] https://github.com/shodanshok/blocksync

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync remote raw block device with --inplace

[Steve Newcomb via rsync]

More notes about diskrsync  :

(0) It seems to work well and efficiently.  The README.md now explains how
to install it and it's a lot easier than the script I included in my last
note would seem to indicate.  (I didn't and still don't know Go.)

(1) Effectively, inherently, and non-optionally, diskrsync has rsync's
--inplace feature. IOW, diskrsync doesn't transfer data redundantly.  Good!

(2) Diskrsync has no bandwidth limiting feature (--bwlimit), nor can the
"trickle" command be used with it in order to limit IP bandwidth usage;
trickle simply has no effect.  Reportedly this is because the Go language
(the "golang" package in Debian-land) uses system calls directly rather
than via libc. :(

(3) Same goes for iotop. Iotop has no effect at launch time, and it can't
re-iotop (reclassify the i/o priorities of) already-running diskrsync
processes, either. :(

(4) The "nice" command doesn't work at diskrsync launch time to control CPU
usage, but renice works on an already-running diskrsync process.

(5) Diskrsync provides no convenient way to affect the processing niceness
or i/o niceness etc. of the process launched at the remote host, as there
is with rsync's --rsync-path feature.

To summarize: while diskrsync is looking like the most suitable solution
known to us, it would still be more convenient, gentler, kinder, and wiser
if the ability to transfer raw block device content were added to rsync.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: Re: rsync remote raw block device with --inplace

[Steve Newcomb via rsync]

These responses have been very useful.  Thanks especially to *Roland*  devzero
at web.de
because...I'm
installing diskrsync  .  So far it
looks like it pretty much works.  It's written in Go, which I hadn't used
before.  There was a bit of a learning curve there, but I attached the
script I used to install it, which is a github-wiki script contributed by
"udif" which I have refined in minor respects.  (Attached.)

I still think this is a reasonable feature for rsync to have, but at least
there appears to be a reasonable solution -- diskrsync -- and I didn't know
about it before.

On Debian Buster/Sid and Debian 9, the following procedure worked, and it
will probably work on Ubuntu and other Debian derivatives.

Note: Be sure to install on both the source and target hosts! If diskrsync
is not found on the remote server, this message appears:

bash: /usr/local/ch-tools3/diskrsync: No such file or directory

Install procedure:

(1) sudo apt-get install golang

(2) As root (if necessary) put the following script in
dir-in-PATH/install_diskrsync.txt (below script contains minor refinements
of udif's script, above).

install_diskrsync.txt


(3) sudo chmod +x dir-in-PATH/install_diskrsync.txt

(4) sudo dir-in-PATH/install_diskrsync.txt

Steve
#!/bin/bash
echo
echo '## '
echo '## installing/updating diskrsync'
echo '## '
echo
set -x

## customize below lines
export GOPATH="/usr/local/ch-static/GO"  ## wherever you keep your go projects
export EXECDIR="/usr/local/ch-tools3"  ## where the executable diskrsync 
command will be put
## customize above lines

export GOBIN="$GOPATH/bin"
export GOROOT="`go env GOROOT`"
export PATH="$PATH:$GOROOT/bin:$GOBIN"
mkdir -p ${GOPATH}/{bin,src,pkg}
mkdir -p ${GOPATH}/src/github.com/dop251

cd ${GOPATH}/src/github.com/dop251

if [ -d diskrsync ] ; then
cd diskrsync
git pull
cd ..
else
git clone https://github.com/dop251/diskrsync.git diskrsync
fi
go get -v github.com/dop251/diskrsync
go build -v github.com/dop251/diskrsync
go install -v github.com/dop251/diskrsync/diskrsync
cp -p ${GOPATH}/bin/diskrsync "$EXECDIR"

set +x
echo
echo '## '
echo '## finished installing/updating diskrsync'
echo '## '
echo
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Aw: Re: Re: rsync remote raw block device with --inplace

[devzero--- via rsync]

maybe this could also be useful:

https://github.com/RyanHow/block2file


> Gesendet: Sonntag, 30. Dezember 2018 um 22:52 Uhr
> Von: "Rolf Fokkens via rsync" 
> An: rsync@lists.samba.org
> Betreff: Re: Aw: Re: rsync remote raw block device with --inplace
>
> It was broucht up before indeed: 
> https://lists.samba.org/archive/rsync/2012-June/027680.html
> 
> On 12/30/18 9:50 PM, devzero--- via rsync wrote:
> >> There have been addons to rsync in the past to do that but rsync really
> >> isn't the correct tool for the job.
> > why not correct tool ?
> >
> > if rsync can greatly keep two large files in sync between source and 
> > destination
> > (using --inplace), why should it (generally spoken) not also be used to 
> > keep two
> > blockdevices in sync ?
> >
> > maybe these links are interesting in that context:
> >
> > https://lists.samba.org/archive/rsync/2010-June/025164.html
> >
> > https://github.com/dop251/diskrsync
> >
> > roland
> >
> >> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
> >> Von: "Kevin Korb via rsync" 
> >> An: rsync@lists.samba.org
> >> Betreff: Re: rsync remote raw block device with --inplace
> >>
> >> There have been addons to rsync in the past to do that but rsync really
> >> isn't the correct tool for the job.  Neither is dd.
> >>
> >> The right tool is something that understands the filesystem within the
> >> block device such as ntfsclone (what I use) or partimage (if you have
> >> ever used Clonezilla this is what it uses).  These will know how to skip
> >> all the empty parts of the filesystem and will still be capable of
> >> restoring a complete image in a bare metal restore.  You can still use
> >> dd to snag a copy of the MBR since that is outside of any filesystems.
> >>
> >> Also, if you do have to resort to a plain image use ddrescue instead of
> >> dd.  It has a status screen and it can resume as long as you used a log
> >> file when you ran it.
> >>
> >> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> >>> It would be very nice to be able to rsync the raw data content of, e.g.,
> >>> a non-mounted disk partition, particularly in combination with --inplace.
> >>>
> >>> Our reality: several dual-boot machines running Windows during the day
> >>> and Linux at night, during backups.  Windows is very tedious and iffy to
> >>> re-reinstall without a raw disk image to start from.  Disks fail, and
> >>> the ensuing downtime must be minimized.
> >>>
> >>> We're using dd for this.  Most of the nightly work is redundant and
> >>> wasteful of elapsed time and storage.  Storage is cheap, but it's not
> >>> *that* cheap.  Elapsed time is priceless.
> >>>
> >>> Rsync refuses to back up raw devices, and even raw character devices,
> >>> with the message "skipping non-regular file" (I think the relevant
> >>> message is in generator.c).
> >>>
> >>> In Linux, anyway, the "raw" command allows a block device to be bound as
> >>> a character device, and then even a "cat" command can read the raw data
> >>> of the block device.  So why does rsync refuse to copy such content, or
> >>> why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
> >>> there are security concerns here, but rsync already disallows some of
> >>> its functions unless the super user is requesting them.
> >>>
> >>>
> >> -- 
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>Kevin Korb  Phone:(407) 252-6853
> >>Systems Administrator   Internet:
> >>FutureQuest, Inc.   ke...@futurequest.net  (work)
> >>Orlando, Floridak...@sanitarium.net (personal)
> >>Web page:   https://sanitarium.net/
> >>PGP public key available on web site.
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>
> >> -- 
> >> Please use reply-all for most replies to avoid omitting the mailing list.
> >> To unsubscribe or change options: 
> >> https://lists.samba.org/mailman/listinfo/rsync
> >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: Re: rsync remote raw block device with --inplace

[Rolf Fokkens via rsync]

It was broucht up before indeed: 
https://lists.samba.org/archive/rsync/2012-June/027680.html


On 12/30/18 9:50 PM, devzero--- via rsync wrote:

There have been addons to rsync in the past to do that but rsync really
isn't the correct tool for the job.

why not correct tool ?

if rsync can greatly keep two large files in sync between source and destination
(using --inplace), why should it (generally spoken) not also be used to keep two
blockdevices in sync ?

maybe these links are interesting in that context:

https://lists.samba.org/archive/rsync/2010-June/025164.html

https://github.com/dop251/diskrsync

roland


Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
Von: "Kevin Korb via rsync" 
An: rsync@lists.samba.org
Betreff: Re: rsync remote raw block device with --inplace

There have been addons to rsync in the past to do that but rsync really
isn't the correct tool for the job.  Neither is dd.

The right tool is something that understands the filesystem within the
block device such as ntfsclone (what I use) or partimage (if you have
ever used Clonezilla this is what it uses).  These will know how to skip
all the empty parts of the filesystem and will still be capable of
restoring a complete image in a bare metal restore.  You can still use
dd to snag a copy of the MBR since that is outside of any filesystems.

Also, if you do have to resort to a plain image use ddrescue instead of
dd.  It has a status screen and it can resume as long as you used a log
file when you ran it.

On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:

It would be very nice to be able to rsync the raw data content of, e.g.,
a non-mounted disk partition, particularly in combination with --inplace.

Our reality: several dual-boot machines running Windows during the day
and Linux at night, during backups.  Windows is very tedious and iffy to
re-reinstall without a raw disk image to start from.  Disks fail, and
the ensuing downtime must be minimized.

We're using dd for this.  Most of the nightly work is redundant and
wasteful of elapsed time and storage.  Storage is cheap, but it's not
*that* cheap.  Elapsed time is priceless.

Rsync refuses to back up raw devices, and even raw character devices,
with the message "skipping non-regular file" (I think the relevant
message is in generator.c).

In Linux, anyway, the "raw" command allows a block device to be bound as
a character device, and then even a "cat" command can read the raw data
of the block device.  So why does rsync refuse to copy such content, or
why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
there are security concerns here, but rsync already disallows some of
its functions unless the super user is requesting them.



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: Re: rsync remote raw block device with --inplace

[Kevin Korb via rsync]

Rsync would only benefit you in terms of reducing network bandwidth
(assuming it is even networking).  The resulting file will be the size
of the block device just like you get with dd[rescue].  You can tell
rsync to write the file sparsely which would mean that any parts of the
disk that have never been written to would probably still contain nulls
and would not be allocated but you will still be copying any deleted
file data that still exists.

Ntfsclone and partimage understand the filesystem.  They will just skip
the blocks that don't contain any current data.  This will keep your
backup file as small as possible.  If you ever have to do a restore they
will put all the file data back where it was and either skip the parts
they didn't save or fill them with nulls.

On 12/30/18 3:50 PM, devz...@web.de wrote:
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.
> 
> why not correct tool ?
> 
> if rsync can greatly keep two large files in sync between source and 
> destination
> (using --inplace), why should it (generally spoken) not also be used to keep 
> two 
> blockdevices in sync ? 
> 
> maybe these links are interesting in that context:
> 
> https://lists.samba.org/archive/rsync/2010-June/025164.html
> 
> https://github.com/dop251/diskrsync
> 
> roland
> 
>> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
>> Von: "Kevin Korb via rsync" 
>> An: rsync@lists.samba.org
>> Betreff: Re: rsync remote raw block device with --inplace
>>
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.  Neither is dd.
>>
>> The right tool is something that understands the filesystem within the
>> block device such as ntfsclone (what I use) or partimage (if you have
>> ever used Clonezilla this is what it uses).  These will know how to skip
>> all the empty parts of the filesystem and will still be capable of
>> restoring a complete image in a bare metal restore.  You can still use
>> dd to snag a copy of the MBR since that is outside of any filesystems.
>>
>> Also, if you do have to resort to a plain image use ddrescue instead of
>> dd.  It has a status screen and it can resume as long as you used a log
>> file when you ran it.
>>
>> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
>>> It would be very nice to be able to rsync the raw data content of, e.g.,
>>> a non-mounted disk partition, particularly in combination with --inplace.
>>>
>>> Our reality: several dual-boot machines running Windows during the day
>>> and Linux at night, during backups.  Windows is very tedious and iffy to
>>> re-reinstall without a raw disk image to start from.  Disks fail, and
>>> the ensuing downtime must be minimized. 
>>>
>>> We're using dd for this.  Most of the nightly work is redundant and
>>> wasteful of elapsed time and storage.  Storage is cheap, but it's not
>>> *that* cheap.  Elapsed time is priceless.
>>>
>>> Rsync refuses to back up raw devices, and even raw character devices,
>>> with the message "skipping non-regular file" (I think the relevant
>>> message is in generator.c). 
>>>
>>> In Linux, anyway, the "raw" command allows a block device to be bound as
>>> a character device, and then even a "cat" command can read the raw data
>>> of the block device.  So why does rsync refuse to copy such content, or
>>> why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
>>> there are security concerns here, but rsync already disallows some of
>>> its functions unless the super user is requesting them. 
>>>
>>>
>>
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>  Kevin Korb  Phone:(407) 252-6853
>>  Systems Administrator   Internet:
>>  FutureQuest, Inc.   ke...@futurequest.net  (work)
>>  Orlando, Floridak...@sanitarium.net (personal)
>>  Web page:   https://sanitarium.net/
>>  PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Ke

Aw: Re: rsync remote raw block device with --inplace

[devzero--- via rsync]

> There have been addons to rsync in the past to do that but rsync really
> isn't the correct tool for the job.

why not correct tool ?

if rsync can greatly keep two large files in sync between source and destination
(using --inplace), why should it (generally spoken) not also be used to keep 
two 
blockdevices in sync ? 

maybe these links are interesting in that context:

https://lists.samba.org/archive/rsync/2010-June/025164.html

https://github.com/dop251/diskrsync

roland

> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
> Von: "Kevin Korb via rsync" 
> An: rsync@lists.samba.org
> Betreff: Re: rsync remote raw block device with --inplace
>
> There have been addons to rsync in the past to do that but rsync really
> isn't the correct tool for the job.  Neither is dd.
> 
> The right tool is something that understands the filesystem within the
> block device such as ntfsclone (what I use) or partimage (if you have
> ever used Clonezilla this is what it uses).  These will know how to skip
> all the empty parts of the filesystem and will still be capable of
> restoring a complete image in a bare metal restore.  You can still use
> dd to snag a copy of the MBR since that is outside of any filesystems.
> 
> Also, if you do have to resort to a plain image use ddrescue instead of
> dd.  It has a status screen and it can resume as long as you used a log
> file when you ran it.
> 
> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> > It would be very nice to be able to rsync the raw data content of, e.g.,
> > a non-mounted disk partition, particularly in combination with --inplace.
> > 
> > Our reality: several dual-boot machines running Windows during the day
> > and Linux at night, during backups.  Windows is very tedious and iffy to
> > re-reinstall without a raw disk image to start from.  Disks fail, and
> > the ensuing downtime must be minimized. 
> > 
> > We're using dd for this.  Most of the nightly work is redundant and
> > wasteful of elapsed time and storage.  Storage is cheap, but it's not
> > *that* cheap.  Elapsed time is priceless.
> > 
> > Rsync refuses to back up raw devices, and even raw character devices,
> > with the message "skipping non-regular file" (I think the relevant
> > message is in generator.c). 
> > 
> > In Linux, anyway, the "raw" command allows a block device to be bound as
> > a character device, and then even a "cat" command can read the raw data
> > of the block device.  So why does rsync refuse to copy such content, or
> > why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
> > there are security concerns here, but rsync already disallows some of
> > its functions unless the super user is requesting them. 
> > 
> > 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync remote raw block device with --inplace

[Kevin Korb via rsync]

There have been addons to rsync in the past to do that but rsync really
isn't the correct tool for the job.  Neither is dd.

The right tool is something that understands the filesystem within the
block device such as ntfsclone (what I use) or partimage (if you have
ever used Clonezilla this is what it uses).  These will know how to skip
all the empty parts of the filesystem and will still be capable of
restoring a complete image in a bare metal restore.  You can still use
dd to snag a copy of the MBR since that is outside of any filesystems.

Also, if you do have to resort to a plain image use ddrescue instead of
dd.  It has a status screen and it can resume as long as you used a log
file when you ran it.

On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> It would be very nice to be able to rsync the raw data content of, e.g.,
> a non-mounted disk partition, particularly in combination with --inplace.
> 
> Our reality: several dual-boot machines running Windows during the day
> and Linux at night, during backups.  Windows is very tedious and iffy to
> re-reinstall without a raw disk image to start from.  Disks fail, and
> the ensuing downtime must be minimized. 
> 
> We're using dd for this.  Most of the nightly work is redundant and
> wasteful of elapsed time and storage.  Storage is cheap, but it's not
> *that* cheap.  Elapsed time is priceless.
> 
> Rsync refuses to back up raw devices, and even raw character devices,
> with the message "skipping non-regular file" (I think the relevant
> message is in generator.c). 
> 
> In Linux, anyway, the "raw" command allows a block device to be bound as
> a character device, and then even a "cat" command can read the raw data
> of the block device.  So why does rsync refuse to copy such content, or
> why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
> there are security concerns here, but rsync already disallows some of
> its functions unless the super user is requesting them. 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html