Re: copy directory tree, mapping to new owners
On 14/09/21 6:50 pm, to...@tuxteam.de wrote: On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: On 13/09/21 7:04 pm, to...@tuxteam.de wrote: >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: >>On 12/09/21 6:52 pm, john doe wrote: > >[...] > >>>If you are doing this in a script, I would use a temporary directory. >>>That way, in case of failure the destination directory is not rongly >>>modified. >>> >>>EG: >>> >>>$ rsync >>> >>>Make the way you want it to be. >>> >>>$ rsync >> >>That is true, but firstly it would require more available space [...] > >This isn't necessary, as you could replace the second `rsync' by a `mv' >(provided your temp tree is on the same storage volume as your target >dir, that is). I was assuming the suggestion was to rsync the source to the temp while the destination still exists, before rsyncing or mv'ing over the top of it. Total of 3 copies (temporarily) rather than 2. Then, it's different. But in your scenario you would probably want to take down whatever "service" relies on the destination dir while the copy is in progress. This is all academic, since rsync with --usermap and --groupmap does what I want, in place. But john doe's proposal had the rationale of "That way, in case of failure the destination directory is not rongly modified." That implies the destination is staying put. Well, I guess deleting it entirely avoids "wrongly modifying" it too :-) In any case, if you haven't the space, you haven't it. Sysadmin's life ain't always nice :) It's available if I want it; everything is on LVM. It's easy to grow. It's also all on a VPS, so expanding the total is a matter of tweaking the settings in a control panel. But it's harder to shrink (especially since I use xfs), so I prefer not to grow it if it's not necessary. Cheers :-) Richard
Re: copy directory tree, mapping to new owners
On Tue 14 Sep 2021 at 08:50:34 (+0200), to...@tuxteam.de wrote: > On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: > > On 13/09/21 7:04 pm, to...@tuxteam.de wrote: > > >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > > >>On 12/09/21 6:52 pm, john doe wrote: > > > > > >[...] > > > > > >>>If you are doing this in a script, I would use a temporary directory. > > >>>That way, in case of failure the destination directory is not rongly > > >>>modified. > > >>> > > >>>EG: > > >>> > > >>>$ rsync > > >>> > > >>>Make the way you want it to be. > > >>> > > >>>$ rsync > > >> > > >>That is true, but firstly it would require more available space [...] > > > > > >This isn't necessary, as you could replace the second `rsync' by a `mv' > > >(provided your temp tree is on the same storage volume as your target > > >dir, that is). > > > > I was assuming the suggestion was to rsync the source to the temp > > while the destination still exists, before rsyncing or mv'ing over > > the top of it. Total of 3 copies (temporarily) rather than 2. > > Then, it's different. But in your scenario you would probably want > to take down whatever "service" relies on the destination dir while > the copy is in progress. As I read the first paragraph of the OP, the service relies on the *source* files, and the destination is just the copy, being created and/or updated for testing. I presume that is why the OP is not too worried by failure. > In any case, if you haven't the space, you haven't it. Sysadmin's > life ain't always nice :) Cheers, David.
Re: copy directory tree, mapping to new owners
On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: > On 13/09/21 7:04 pm, to...@tuxteam.de wrote: > >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > >>On 12/09/21 6:52 pm, john doe wrote: > > > >[...] > > > >>>If you are doing this in a script, I would use a temporary directory. > >>>That way, in case of failure the destination directory is not rongly > >>>modified. > >>> > >>>EG: > >>> > >>>$ rsync > >>> > >>>Make the way you want it to be. > >>> > >>>$ rsync > >> > >>That is true, but firstly it would require more available space [...] > > > >This isn't necessary, as you could replace the second `rsync' by a `mv' > >(provided your temp tree is on the same storage volume as your target > >dir, that is). > > I was assuming the suggestion was to rsync the source to the temp > while the destination still exists, before rsyncing or mv'ing over > the top of it. Total of 3 copies (temporarily) rather than 2. Then, it's different. But in your scenario you would probably want to take down whatever "service" relies on the destination dir while the copy is in progress. In any case, if you haven't the space, you haven't it. Sysadmin's life ain't always nice :) Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
On 13/09/21 7:04 pm, to...@tuxteam.de wrote: On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: On 12/09/21 6:52 pm, john doe wrote: [...] >If you are doing this in a script, I would use a temporary directory. >That way, in case of failure the destination directory is not rongly >modified. > >EG: > >$ rsync > >Make the way you want it to be. > >$ rsync That is true, but firstly it would require more available space [...] This isn't necessary, as you could replace the second `rsync' by a `mv' (provided your temp tree is on the same storage volume as your target dir, that is). I was assuming the suggestion was to rsync the source to the temp while the destination still exists, before rsyncing or mv'ing over the top of it. Total of 3 copies (temporarily) rather than 2. Cheers, Richard
Re: copy directory tree, mapping to new owners
On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > On 12/09/21 6:52 pm, john doe wrote: [...] > >If you are doing this in a script, I would use a temporary directory. > >That way, in case of failure the destination directory is not rongly > >modified. > > > >EG: > > > >$ rsync > > > >Make the way you want it to be. > > > >$ rsync > > That is true, but firstly it would require more available space [...] This isn't necessary, as you could replace the second `rsync' by a `mv' (provided your temp tree is on the same storage volume as your target dir, that is). Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
On 12/09/21 7:46 pm, Teemu Likonen wrote: * 2021-09-12 12:43:29+1200, Richard Hector wrote: The context of my question is that I'm creating (or updating) a test copy of a website. The files are owned by one of two owners, depending on whether they were written by the server (actually php-fpm). To do that, I want all the permissions to remain the same, but the ownership should be changed according to a provided map. Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also "--groupmap" option for mapping groups. Aha - thanks a lot :-) I guess I've been caught not reading the man page thoroughly enough. The pitfalls of thinking that you know everything that a tool does, just because you use it often ... The way the docs are written seems to imply a sender and receiver; I'll have to check it works for a local copy ... it does. Now I need to rewrite my script somewhat, since to do this it's going to have to run as root ... Thanks, Richard
Re: copy directory tree, mapping to new owners
On 12/09/21 6:53 pm, l0f...@tuta.io wrote: # actually not necessary? rsync will create it mkdir -p mysite_test/doc_root You can make a simple test to know that but I would say that rsync doesn't create your destination "root" directory (the one you specify on the command line) unless `--mkpath` is used. I was fairly confident of that, from my regular usage. # The trailing / matters. Does it matter on the source as well? # I generally include it. rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. Actually, I'm not sure to understand Greg's remark here. In my opinion, trailing slash doesn't matter for destination folder on the contrary of *source* folder. In other words, for me, the following are equal: rsync -a mysite/doc_root/ mysite_test/doc_root/ rsync -a mysite/doc_root/ mysite_test/doc_root But not the following: rsync -a mysite/doc_root mysite_test/doc_root => you will get an extra "doc_root" folder (the source one) in your dest, i.e. : mysite_test/doc_root/doc_root and then the content of doc_root source rsync -a mysite/doc_root/ mysite_test/doc_root => your doc_root (destination) folder will get doc_root content (source) directly Yep. As I've mentioned elsewhere, I habitually include trailing slashes for both source and destination when I'm replicating a whole tree. I couldn't remember the details of what happens when you don't, but I know what happens when I do :-) Cheers, Richard
Re: copy directory tree, mapping to new owners
On 12/09/21 6:52 pm, john doe wrote:
On 9/12/2021 3:45 AM, Richard Hector wrote:
Thanks, that looks reasonable. It does mean, though, that the files
exist for a while with the wrong ownership. That probably doesn't
matter, but somehow 'feels wrong' to me.
If you are doing this in a script, I would use a temporary directory.
That way, in case of failure the destination directory is not rongly
modified.
EG:
$ rsync
Make the way you want it to be.
$ rsync
That is true, but firstly it would require more available space, and
secondly, as long as I know about the failure, it doesn't worry me too
much. This is a script I will call manually.
Given that you want to change the ownership, you may want to emulate the
options implied by '-a' but without the ownership option ('-o').
I do that in my existing script (also not -g (group) or -D
(specials/devices (which don't exist there anyway)) - and sometimes -p
(permissions), if I've pre-created the tree with the permissions I want.
The last one is a slightly different use case, but I should make sure I
know which use case I'm trying for ...
My habits would also lead me to do all of the above from above the two
directories in question (in my case, /srv - for /srv/mysite/doc_root and
/srv/mysite_test/doc_root) So:
cd /srv
# actually not necessary? rsync will create it
mkdir -p mysite_test/doc_root
# The trailing / matters. Does it matter on the source as well?
According to the man page it does (1):
"CWrsync -avz foo:src/bar/ /data/tmp
A trailing slash on the source changes this behavior to avoid creating
an additional directory level at the destination. You can think of a
trailing / on a source as meaning lqcopy the contents"
# I generally include it.
In a script, you need to be sure what the cmd does do or not do!!! :)
True. But many options I've learned from my everyday usage, and
sometimes forgotten the rationale - I just know it does what I want :-)
rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing /
matters.
find mysite_dest -user mysite -exec chown mysite_test {} +
# I prefer mysite_test-run; it keeps consistency with
# the ${sitename}-run pattern used elsewhere
find mysite_dest -user mysite-run -exec chown mysite_test-run {} +
Have I broken anything there?
Not that I can see but testing will tell!
This is what I would do. And I would do it *interactively*.
If you insist on making a script, then it will be slightly more
complicated, because you'll need to add error checking.
The trouble with doing it interactively, when it needs to be done many
times (and on several sites), is that each time there's opportunity to
make a mistake. And it means the process needs to be documented
separately from the script.
In fact, I'd incorporate the above in a larger script, which does things
like copying the database (and changing the db name in the site config).
Error checking in shell scripts is something I certainly need to learn
At the very least, ' ['&&'|'||'] handle the error>.
If you use Bash and don't need to be POSIX compliant and /or portable
you can be a bit more creative.
Heh - you've cut my sentence in two, which appears to change the meaning
slightly :-)
" ... is something I need to learn and practice more".
You make it look like I need to learn error checking from scratch :-)
I am using bash. And I am currently using 'set -e' to handle that kind
of error globally. As I say, as long as I can see that it broke, it
doesn't matter too much.
and practice more - I tend to rely somewhat on 'knowing' what I'm
working with, which is probably not a good idea.
>
Yes in a script it is like shooting yourself in the foot.
I do test it.
Cheers,
Richard
Re: copy directory tree, mapping to new owners
Hi Curt, 12 sept. 2021, 16:23 de cu...@free.fr: > On 2021-09-12, l0f...@tuta.io wrote: > >> In my opinion, trailing slash doesn't matter for destination folder on >> the contrary of *source* folder. >> > It makes a difference when the source is a file and the destination > directory is inexistant, because in that case a copy of $file named > $destination will be created, which is not what you want here. > I think it is probably not applicable here for the OP (it's a directory) but thanks for this special use case :) This explanation is given in the aforementioned `--mkpath` switch by the way. The moral is: Pay attention to the single one character you type on the command line! Best regards, l0f4r0
Re: copy directory tree, mapping to new owners
On 2021-09-12, l0f...@tuta.io wrote: >> # The trailing / matters. Does it matter on the source as well? >> # I generally include it. >> rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / >> matters. >> > Actually, I'm not sure to understand Greg's remark here. > > In my opinion, trailing slash doesn't matter for destination folder on > the contrary of *source* folder. > It makes a difference when the source is a file and the destination directory is inexistant, because in that case a copy of $file named $destination will be created, which is not what you want here.
Re: copy directory tree, mapping to new owners
On Du, 12 sep 21, 08:52:55, john doe wrote:
> On 9/12/2021 3:45 AM, Richard Hector wrote:
> > On 12/09/21 12:52 pm, Greg Wooledge wrote:
> > >
> > > cd /src
> > > mkdir -p /dest
> > > rsync -a . /dest/ # The trailing / matters.
> > > cd /dest
> > > find . -user mysite -exec chown mysite_test {} +
> > > find . -user mysite-run -exec chown mysite-run_test {} +
>
> Given that you want to change the ownership, you may want to emulate the
> options implied by '-a' but without the ownership option ('-o').
It might not matter for a site[1], but do note that -a doesn't include
-H (preserve hardlinks), -A (preserve ACLs) and -X (preserve extended
attributes).
[1] It definitely matters for system installations, as I found out the
hard way.
Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser
signature.asc
Description: PGP signature
Re: copy directory tree, mapping to new owners
On Sun, Sep 12, 2021 at 10:46:01AM +0300, Teemu Likonen wrote: [...] > Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also > "--groupmap" option for mapping groups. Never underestimate rsync :-) Actually, I do read its manpage from time to time. Should do it more often, perhaps as (yet another) after-breakfast routine. Thanks, Teemu, this one wasn't aware to me! Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
* 2021-09-12 12:43:29+1200, Richard Hector wrote: > The context of my question is that I'm creating (or updating) a test > copy of a website. The files are owned by one of two owners, depending > on whether they were written by the server (actually php-fpm). > > To do that, I want all the permissions to remain the same, but the > ownership should be changed according to a provided map. Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also "--groupmap" option for mapping groups. -- /// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/ // OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450 signature.asc Description: PGP signature
Re: copy directory tree, mapping to new owners
12 sept. 2021, 08:53 de l0f...@tuta.io:
> 12 sept. 2021, 03:45 de rich...@walnut.gen.nz:
>
>> # actually not necessary? rsync will create it
>> mkdir -p mysite_test/doc_root
>>
> You can make a simple test to know that but I would say that rsync doesn't
> create your destination "root" directory (the one you specify on the command
> line) unless `--mkpath` is used.
>
Actually, I've just did some tests and it appears that rsync creates the
destination folder if its parent already exists and if you have the right
permissions.
For example ("my" and "dest" don't exist yet):
rsync -a src/ dest => "dest" is created
rsync -a src/ my/dest => error, "my" and "dest" don't exist (and are not
created then)
rsync --mkpath -a src/ my/dest => "my" and "dest" are created
l0f4r0
Re: copy directory tree, mapping to new owners
Hi,
12 sept. 2021, 03:45 de rich...@walnut.gen.nz:
> On 12/09/21 12:52 pm, Greg Wooledge wrote:
>
>> cd /src
>> mkdir -p /dest
>> rsync -a . /dest/ # The trailing / matters.
>> cd /dest
>> find . -user mysite -exec chown mysite_test {} +
>> find . -user mysite-run -exec chown mysite-run_test {} +
>>
>
> [...]
> # actually not necessary? rsync will create it
> mkdir -p mysite_test/doc_root
>
You can make a simple test to know that but I would say that rsync doesn't
create your destination "root" directory (the one you specify on the command
line) unless `--mkpath` is used.
> # The trailing / matters. Does it matter on the source as well?
> # I generally include it.
> rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters.
>
Actually, I'm not sure to understand Greg's remark here.
In my opinion, trailing slash doesn't matter for destination folder on the
contrary of *source* folder.
In other words, for me, the following are equal:
rsync -a mysite/doc_root/ mysite_test/doc_root/
rsync -a mysite/doc_root/ mysite_test/doc_root
But not the following:
rsync -a mysite/doc_root mysite_test/doc_root => you will get an extra
"doc_root" folder (the source one) in your dest, i.e. :
mysite_test/doc_root/doc_root and then the content of doc_root source
rsync -a mysite/doc_root/ mysite_test/doc_root => your doc_root (destination)
folder will get doc_root content (source) directly
Best regards,
l0f4r0
Re: copy directory tree, mapping to new owners
On 9/12/2021 3:45 AM, Richard Hector wrote:
On 12/09/21 12:52 pm, Greg Wooledge wrote:
On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote:
The context of my question is that I'm creating (or updating) a test
copy of
a website. The files are owned by one of two owners, depending on
whether
they were written by the server (actually php-fpm).
To do that, I want all the permissions to remain the same, but the
ownership
should be changed according to a provided map. For example, if the
old file
was owned by 'mysite', the copy should be owned by 'mysite_test'. If
the old
file was owned by 'mysite-run' (the user php runs as), the copy
should be
owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make
things easier, I can live with that).
cd /src
mkdir -p /dest
rsync -a . /dest/ # The trailing / matters.
cd /dest
find . -user mysite -exec chown mysite_test {} +
find . -user mysite-run -exec chown mysite-run_test {} +
Thanks, that looks reasonable. It does mean, though, that the files
exist for a while with the wrong ownership. That probably doesn't
matter, but somehow 'feels wrong' to me.
If you are doing this in a script, I would use a temporary directory.
That way, in case of failure the destination directory is not rongly
modified.
EG:
$ rsync
Make the way you want it to be.
$ rsync
Given that you want to change the ownership, you may want to emulate the
options implied by '-a' but without the ownership option ('-o').
My habits would also lead me to do all of the above from above the two
directories in question (in my case, /srv - for /srv/mysite/doc_root and
/srv/mysite_test/doc_root) So:
cd /srv
# actually not necessary? rsync will create it
mkdir -p mysite_test/doc_root
# The trailing / matters. Does it matter on the source as well?
According to the man page it does (1):
"CWrsync -avz foo:src/bar/ /data/tmp
A trailing slash on the source changes this behavior to avoid creating
an additional directory level at the destination. You can think of a
trailing / on a source as meaning lqcopy the contents"
# I generally include it.
In a script, you need to be sure what the cmd does do or not do!!! :)
rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing /
matters.
find mysite_dest -user mysite -exec chown mysite_test {} +
# I prefer mysite_test-run; it keeps consistency with
# the ${sitename}-run pattern used elsewhere
find mysite_dest -user mysite-run -exec chown mysite_test-run {} +
Have I broken anything there?
Not that I can see but testing will tell!
This is what I would do. And I would do it *interactively*.
If you insist on making a script, then it will be slightly more
complicated, because you'll need to add error checking.
The trouble with doing it interactively, when it needs to be done many
times (and on several sites), is that each time there's opportunity to
make a mistake. And it means the process needs to be documented
separately from the script.
In fact, I'd incorporate the above in a larger script, which does things
like copying the database (and changing the db name in the site config).
Error checking in shell scripts is something I certainly need to learn
At the very least, ' ['&&'|'||'] handle the error>.
If you use Bash and don't need to be POSIX compliant and /or portable
you can be a bit more creative.
and practice more - I tend to rely somewhat on 'knowing' what I'm
working with, which is probably not a good idea.
>
Yes in a script it is like shooting yourself in the foot.
1) https://linux.die.net/man/1/rsync
--
John Doe
Re: copy directory tree, mapping to new owners
On 12/09/21 12:52 pm, Greg Wooledge wrote:
On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote:
The context of my question is that I'm creating (or updating) a test copy of
a website. The files are owned by one of two owners, depending on whether
they were written by the server (actually php-fpm).
To do that, I want all the permissions to remain the same, but the ownership
should be changed according to a provided map. For example, if the old file
was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old
file was owned by 'mysite-run' (the user php runs as), the copy should be
owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make
things easier, I can live with that).
cd /src
mkdir -p /dest
rsync -a . /dest/ # The trailing / matters.
cd /dest
find . -user mysite -exec chown mysite_test {} +
find . -user mysite-run -exec chown mysite-run_test {} +
Thanks, that looks reasonable. It does mean, though, that the files
exist for a while with the wrong ownership. That probably doesn't
matter, but somehow 'feels wrong' to me.
My habits would also lead me to do all of the above from above the two
directories in question (in my case, /srv - for /srv/mysite/doc_root and
/srv/mysite_test/doc_root) So:
cd /srv
# actually not necessary? rsync will create it
mkdir -p mysite_test/doc_root
# The trailing / matters. Does it matter on the source as well?
# I generally include it.
rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing /
matters.
find mysite_dest -user mysite -exec chown mysite_test {} +
# I prefer mysite_test-run; it keeps consistency with
# the ${sitename}-run pattern used elsewhere
find mysite_dest -user mysite-run -exec chown mysite_test-run {} +
Have I broken anything there?
This is what I would do. And I would do it *interactively*.
If you insist on making a script, then it will be slightly more
complicated, because you'll need to add error checking.
The trouble with doing it interactively, when it needs to be done many
times (and on several sites), is that each time there's opportunity to
make a mistake. And it means the process needs to be documented
separately from the script.
In fact, I'd incorporate the above in a larger script, which does things
like copying the database (and changing the db name in the site config).
Error checking in shell scripts is something I certainly need to learn
and practice more - I tend to rely somewhat on 'knowing' what I'm
working with, which is probably not a good idea.
Thanks,
Richard
Re: copy directory tree, mapping to new owners
On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote:
> The context of my question is that I'm creating (or updating) a test copy of
> a website. The files are owned by one of two owners, depending on whether
> they were written by the server (actually php-fpm).
>
> To do that, I want all the permissions to remain the same, but the ownership
> should be changed according to a provided map. For example, if the old file
> was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old
> file was owned by 'mysite-run' (the user php runs as), the copy should be
> owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make
> things easier, I can live with that).
cd /src
mkdir -p /dest
rsync -a . /dest/ # The trailing / matters.
cd /dest
find . -user mysite -exec chown mysite_test {} +
find . -user mysite-run -exec chown mysite-run_test {} +
This is what I would do. And I would do it *interactively*.
If you insist on making a script, then it will be slightly more
complicated, because you'll need to add error checking.
