Re: Aw: Re: rsync remote raw block device with --inplace

[Steve Newcomb via rsync]

These responses have been very useful.  Thanks especially to *Roland*  devzero
at web.de
because...I'm
installing diskrsync  .  So far it
looks like it pretty much works.  It's written in Go, which I hadn't used
before.  There was a bit of a learning curve there, but I attached the
script I used to install it, which is a github-wiki script contributed by
"udif" which I have refined in minor respects.  (Attached.)

I still think this is a reasonable feature for rsync to have, but at least
there appears to be a reasonable solution -- diskrsync -- and I didn't know
about it before.

On Debian Buster/Sid and Debian 9, the following procedure worked, and it
will probably work on Ubuntu and other Debian derivatives.

Note: Be sure to install on both the source and target hosts! If diskrsync
is not found on the remote server, this message appears:

bash: /usr/local/ch-tools3/diskrsync: No such file or directory

Install procedure:

(1) sudo apt-get install golang

(2) As root (if necessary) put the following script in
dir-in-PATH/install_diskrsync.txt (below script contains minor refinements
of udif's script, above).

install_diskrsync.txt


(3) sudo chmod +x dir-in-PATH/install_diskrsync.txt

(4) sudo dir-in-PATH/install_diskrsync.txt

Steve
#!/bin/bash
echo
echo '## '
echo '## installing/updating diskrsync'
echo '## '
echo
set -x

## customize below lines
export GOPATH="/usr/local/ch-static/GO"  ## wherever you keep your go projects
export EXECDIR="/usr/local/ch-tools3"  ## where the executable diskrsync 
command will be put
## customize above lines

export GOBIN="$GOPATH/bin"
export GOROOT="`go env GOROOT`"
export PATH="$PATH:$GOROOT/bin:$GOBIN"
mkdir -p ${GOPATH}/{bin,src,pkg}
mkdir -p ${GOPATH}/src/github.com/dop251

cd ${GOPATH}/src/github.com/dop251

if [ -d diskrsync ] ; then
cd diskrsync
git pull
cd ..
else
git clone https://github.com/dop251/diskrsync.git diskrsync
fi
go get -v github.com/dop251/diskrsync
go build -v github.com/dop251/diskrsync
go install -v github.com/dop251/diskrsync/diskrsync
cp -p ${GOPATH}/bin/diskrsync "$EXECDIR"

set +x
echo
echo '## '
echo '## finished installing/updating diskrsync'
echo '## '
echo
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync remote raw block device with --inplace

[Steve Newcomb via rsync]

More notes about diskrsync  :

(0) It seems to work well and efficiently.  The README.md now explains how
to install it and it's a lot easier than the script I included in my last
note would seem to indicate.  (I didn't and still don't know Go.)

(1) Effectively, inherently, and non-optionally, diskrsync has rsync's
--inplace feature. IOW, diskrsync doesn't transfer data redundantly.  Good!

(2) Diskrsync has no bandwidth limiting feature (--bwlimit), nor can the
"trickle" command be used with it in order to limit IP bandwidth usage;
trickle simply has no effect.  Reportedly this is because the Go language
(the "golang" package in Debian-land) uses system calls directly rather
than via libc. :(

(3) Same goes for iotop. Iotop has no effect at launch time, and it can't
re-iotop (reclassify the i/o priorities of) already-running diskrsync
processes, either. :(

(4) The "nice" command doesn't work at diskrsync launch time to control CPU
usage, but renice works on an already-running diskrsync process.

(5) Diskrsync provides no convenient way to affect the processing niceness
or i/o niceness etc. of the process launched at the remote host, as there
is with rsync's --rsync-path feature.

To summarize: while diskrsync is looking like the most suitable solution
known to us, it would still be more convenient, gentler, kinder, and wiser
if the ability to transfer raw block device content were added to rsync.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync remote raw block device with --inplace

[Steve Newcomb via rsync]

It would be very nice to be able to rsync the raw data content of, e.g., a
non-mounted disk partition, particularly in combination with --inplace.

Our reality: several dual-boot machines running Windows during the day and
Linux at night, during backups.  Windows is very tedious and iffy to
re-reinstall without a raw disk image to start from.  Disks fail, and the
ensuing downtime must be minimized.

We're using dd for this.  Most of the nightly work is redundant and
wasteful of elapsed time and storage.  Storage is cheap, but it's not
*that* cheap.  Elapsed time is priceless.

Rsync refuses to back up raw devices, and even raw character devices, with
the message "skipping non-regular file" (I think the relevant message is in
generator.c).

In Linux, anyway, the "raw" command allows a block device to be bound as a
character device, and then even a "cat" command can read the raw data of
the block device.  So why does rsync refuse to copy such content, or why is
it a bad idea, or what rsync doctrine conflicts with it?  I agree there are
security concerns here, but rsync already disallows some of its functions
unless the super user is requesting them.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html